Cooperative minimally invasive telesurgical system

ABSTRACT

Improved robotic surgical systems, devices, and methods include selectably associatable master/slave pairs, often having more manipulator arms than will be moved simultaneously by the two hands of a surgeon. Four manipulator arms can support an image capture device, a left hand tissue manipulation tool, a right hand tissue manipulation tool, and a fourth surgical instrument, particularly for stabilizing, retracting, tool change, or other functions benefiting from intermittent movement. The four or more arms may sequentially be controlled by left and right master input control devices. The fourth arm may be used to support another image capture device, and control of some or all of the arms may be transferred back-and-forth between the operator and an assistant. Two or more robotic systems each having master controls and slave manipulators may be coupled to enable cooperative surgery between two or more operators.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of application Ser. No.09/433,120 (Attorney Docket No. 017516-004720), filed Nov. 3, 1999, fora “Cooperative Minimally Invasive Telesurgical System”, which was acontinuation-in-part of and claims the benefit of priority fromapplication Ser. No. 09/399,457 (Attorney Docket No. 017516-004710),filed Sep. 17, 1999 for a “Cooperative Minimally Invasive TelesurgicalSystem”; application Ser. No. 09/374,643 (Attorney Docket No.017516-005900) (Abandoned), filed Aug. 16, 1999 for a “CooperativeMinimally Invasive Telesurgical System”; and also claims the benefit ofpriority from Provisional Application Ser. No. 60/116,891 (AttorneyDocket No. 017516-004700), filed Jan. 22, 1999, for “Dynamic Associationof Master and Slave in a Minimally Invasive Telesurgical System”;Provisional Application Ser. No. 60/116,842 (Attorney Docket No.017516-001300), filed Jan. 22, 1999, for “Repositioning andReorientation of Master/Slave Relationship in Minimally InvasiveTelesurgery”; and Provisional Application Ser. No. 60/109,359 (AttorneyDocket No. 017516-002500), filed Nov. 20, 1998, for “Apparatus andMethod for Tracking and Controlling Cardiac Motion During CardiacSurgery Without Cardioplegia”; the full disclosures of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

The present application is generally directed to medical devices,systems, and methods. In a particular embodiment, the invention providestelesurgical robotic systems and methods that flexibly and selectablycouple input devices to robotic manipulator arms during surgery.

Advances in minimally invasive surgical technology could dramaticallyincrease the number of surgeries performed in a minimally invasivemanner. Minimally invasive medical techniques are aimed at reducing theamount of extraneous tissue that is damaged during diagnostic orsurgical procedures, thereby reducing patient recovery time, discomfort,and deleterious side effects. The average length of a hospital stay fora standard surgery may also be shortened significantly using minimallyinvasive surgical techniques. Thus, an increased adoption of minimallyinvasive techniques could save millions of hospital days, and millionsof dollars annually in hospital residency costs alone. Patient recoverytimes, patient discomfort, surgical side effects, and time away fromwork may also be reduced with minimally invasive surgery.

The most common form of minimally invasive surgery may be endoscopy.Probably the most common form of endoscopy is laparoscopy, which isminimally invasive inspection and surgery inside the abdominal cavity.In standard laparoscopic surgery, a patient's abdomen is insufflatedwith gas, and cannula sleeves are passed through small (approximately ½inch or less) incisions to provide entry ports for laparoscopic surgicalinstruments. The laparoscopic surgical instruments generally include alaparoscope (for viewing the surgical field) and working tools. Theworking tools are similar to those used in conventional (open) surgery,except that the working end or end effector of each tool is separatedfrom its handle by an extension tube. As used herein, the term “endeffector” means the actual working part of the surgical instrument andcan include clamps, graspers, scissors, staplers, image capture lenses,and needle holders, for example. To perform surgical procedures, thesurgeon passes these working tools or instruments through the cannulasleeves to an internal surgical site and manipulates them from outsidethe abdomen. The surgeon monitors the procedure by means of a monitorthat displays an image of the surgical site taken from the laparoscope.Similar endoscopic techniques are employed in, e.g., arthroscopy,retroperitoneoscopy, pelviscopy, nephroscopy, cystoscopy, cisternoscopy,sinoscopy, hysteroscopy, urethroscopy, and the like.

There are many disadvantages relating to current minimally invasivesurgical (MIS) technology. For example, existing MIS instruments denythe surgeon the flexibility of tool placement found in open surgery.Most current laparoscopic tools have rigid shafts, so that it can bedifficult to approach the worksite through the small incision.Additionally, the length and construction of many endoscopic instrumentsreduces the surgeon's ability to feel forces exerted by tissues andorgans on the end effector of the associated tool. The lack of dexterityand sensitivity of endoscopic tools is a major impediment to theexpansion of minimally invasive surgery.

Minimally invasive telesurgical robotic systems are being developed toincrease a surgeon's dexterity when working within an internal surgicalsite, as well as to allow a surgeon to operate on a patient from aremote location. In a telesurgery system, the surgeon is often providedwith an image of the surgical site at a computer workstation. Whileviewing a three-dimensional image of the surgical site on a suitableviewer or display, the surgeon performs the surgical procedures on thepatient by manipulating master input or control devices of theworkstation. The master controls the motion of a servomechanicallyoperated surgical instrument. During the surgical procedure, thetelesurgical system can provide mechanical actuation and control of avariety of surgical instruments or tools having end effectors such as,e.g., tissue graspers, needle drivers, or the like, that perform variousfunctions for the surgeon, e.g., holding or driving a needle, grasping ablood vessel, or dissecting tissue, or the like, in response tomanipulation of the master control devices.

While the proposed robotic surgery systems offer significant potentialto increase the number of procedures that can be performed in aminimally invasive manner, still further improvements are desirable. Inparticular, previous proposals for robotic surgery often emphasizedirect replacement of the mechanical connection between the handles andend effectors of known minimally invasive surgical tools with a roboticservomechanism. Work in connection with the present invention suggeststhat integration of robotic capabilities into the operating theater canbenefit from significant changes to this one-to-one replacement model.Realization of the full potential of robotically assisted surgery mayinstead benefit from significant revisions to the interactions and rolesof team members, as compared to the roles performed by surgical teammembers during open and known minimally invasive surgical procedures.

In light of the above, it would be beneficial to provide improvedrobotic surgical devices, systems, and methods for performing roboticsurgery. It would be beneficial if these improved techniques enhancedthe overall capabilities of telesurgery by recognizing, accommodating,and facilitating the new roles that may be performed by the team membersof a robotic surgical team. It would further be beneficial if theseimprovements facilitated complex robotic surgeries such as coronaryartery bypass grafting, particularly while minimizing the total numberof personnel (and hence the expense) involved in these roboticprocedures. It would be best if these benefits could be provided whileenhancing the overall control over the surgical instruments and safetyof the surgical procedure, while avoiding excessive complexity andredundancy in the robotic system. Some or all of these advantages areprovided by the invention described hereinbelow.

SUMMARY OF THE INVENTION

The present invention provides improved robotic surgical systems,devices, and methods. The robotic surgical systems of the presentinvention will often include a plurality of input devices and/or aplurality of robotic manipulator arms for moving surgical instruments. Aprocessor will often selectably couple a selected input device to aselected manipulator arm, and allows modification of the operativeassociation so that that same input device can be coupled to a differentmanipulator arm, and/or so that that same manipulator arm can becontrolled by a different input device. This selective coupling, forexample, allows the controller to properly assign left and rightsurgical end effectors to left and right input devices for use by anoperator viewing the procedure via an image capture device. When theimage capture devices moves, the operative associations can be revised.In some embodiments, the image capture device may be removed from one ofthe manipulator arms and instead mounted to another of the manipulatorarms, with the left and right input devices reassigned so as to avoid anawkward surgical environment for the system operator sitting at a mastercontrol station.

The systems of the present invention will often include more manipulatorarms than will be moved simultaneously by a single surgeon. In additionto an imaging arm (movably supporting an image capture device) and twomanipulator arms (holding selectably designated “left” and “right”surgical tools for manipulation by left and right hands of the systemoperator, e.g.), one or more additional manipulator arms will often beprovided to position associated surgical instrument(s). At least one ofthe additional manipulator arms may be maintained in a stationaryconfiguration to stabilize or retract tissue while the operator movesleft and right input devices with his or her left and right hands tomanipulate tissues with the associated surgical tools. The one or moreadditional arms may also be used to support another image capturedevice, often a second endoscope to view the internal surgical site froman alternative vantage point. Additional arms may optionally provide oneor more additional surgical tools for manipulating tissue or aiding inthe performance of a procedure at a surgical site. To manipulate thisadditional surgical instrument, an assistant input device may optionallybe provided so that an assistant (such as an assisting surgeon atanother workstation, a surgical nurse at the patient's side, or thelike) can position the additional robotic arm(s). Regardless of thepresence or absence of such an assistant input device, the roboticsurgical systems of the present invention will preferably allow thesystem operator to selectively associate the right and/or left inputdevices of the surgical workstation with any of a plurality of surgicalinstruments.

In a first aspect, the invention provides a robotic surgical systemcomprising a first input device manipulatable by a hand of an operator.A first robotic arm assembly includes a first manipulator arm for movinga first surgical instrument. A second robotic arm assembly includes asecond manipulator arm for moving a second surgical instrument. Acontrol system couples the first input device to the first and secondrobotic arm assemblies. The control system permits selective operativeassociation of the first input device with the first robotic armassembly, and also permits selective operative association of the firstinput device with the second robotic arm assembly.

Typically, the control system will have a plurality of selectable modes,with manipulation of the first input device effecting correspondingmovement of the first surgical instrument in one mode, and with the samemanipulation of the first input device effecting corresponding movementof the second surgical instrument in a second mode. In some embodiments,a second input device may be used to effect corresponding movement ofthe second surgical instrument when the control system is in the firstmode, and effect corresponding movement of the first surgical instrumentwhen the control system is in the second mode, allowing swapping controlof the surgical instruments between the input devices. This isparticularly useful when the system operator is controlling the firstand second input devices using left and right hands with reference to animage of an internal surgical site, as it allows the system operator toswitch tools when the image capture device providing the image moves towhat would otherwise be an awkward position.

In another aspect, the invention provides a robotic surgical systemcomprising a plurality of input devices and a plurality of manipulatorarms, each manipulator arm having an instrument holder. A plurality ofsurgical instruments are mountable to the instrument holders, thesurgical instruments including an image capture device and a tool havinga surgical end effector for treating tissue. A control system couplesthe input devices with the manipulator arms. The control systemselectably associates each input device with a manipulator arm.

In some embodiments, the image capture device may be removed from onemanipulator arm and mounted to an alternative manipulator arm, oftenwith the control system being reconfigurable so that the input devicesare operatively associatable with manipulator arms holding tools fortreating tissue. A particularly advantageous control system is providedwhich allows this flexible pairing of input devices and manipulatorarms.

In a specific aspect, the invention provides a minimally invasiverobotic surgical system comprising two input devices and at least twomedical instrument robotic arm assemblies. One of the input devices isoperatively associated with one of the robotic arm assemblies to causemovement of the robotic arm assembly in response to inputs on the inputdevice. The other input device is operatively associated with another ofthe robotic arm assemblies to cause movement of that other robotic armassembly in response to inputs on that other input device. A controlsystem couples the input devices with the robotic aim assemblies. Thecontrol system enables selective swapping so as to cause the inputdevice to be operatively associated with the robotic arm assembly whichwas operatively associated with the other input device, and to cause theother input device to be operatively associated with the robotic armassembly which was operatively associated with the input device. Relatedsystems allow selective operative association between at least one ofthe input devices and an image capture robotic arm assembly to permitthe position of an image capture device to be changed using the at leastone input device.

In a method aspect, the invention provides a robotic surgical methodcomprising robotically moving a first surgical instrument using a firstmanipulator arm by manipulating a first input device with a hand. Acontrol system is reconfigured by entering a command. The control systemcouples the first input device with the first manipulator aim, and alsowith a second manipulator arm. A second surgical instrument is movedrobotically using the second manipulator arm by manipulating the inputdevice with the hand after the reconfiguring step.

In a related method, a robotic surgical method comprises roboticallymoving a surgical instrument using a manipulator arm by manipulating afirst input device with a first hand. A control system is reconfiguredby entering a command. The control system couples the first inputdevice, and also a second input device, with the manipulator arm. Thesurgical instrument is robotically moved using the manipulator arm bymanipulating the second input device with a second hand after thereconfiguring step.

In another aspect, the invention provides a robotic surgical systemcomprising an imaging system transmitting an image from an image capturedevice to a display. First, second, and third manipulator arms eachsupport an associated surgical instrument. A master controller isdisposed adjacent the display. The master has a first input devicemanipulatable by a first hand of an operator, and a second input devicemanipulatable by a second hand of the operator. A processor operativelycouples each input device of the master to an associated manipulator armso that movement of the input device effects movement of the associatedsurgical instrument.

In some embodiments, the surgical instrument associated with the thirdarm will comprise another image capture device. Where two image capturedevices are included in the system, the processor will preferablytransmit arm movement command signals to the arms according to differentcoordinate system transformations depending on which image capturedevice is providing the image currently shown on the display. Thisallows the processor to correlate between a direction of movement of theinput device and the movement of the surgical instrument when switchingbetween two different endoscopes having different fields of view.

Preferably, at least the third robotic instrument arm should beconfigurable to maintain a stationary configuration under somecircumstances, with the arm in the stationary configuration inhibitingmovement of the associated surgical instrument despite movement of theinput devices. Such a stationary configuration is particularly usefulwhen the surgical instrument mounted on the third arm comprises astabilizer (such as a coronary tissue stabilizer used for beating heartsurgery) or a retractor (for example, to retract tissue to expose adesired area of the cystic duct to the surgeon during cholecystectomy).The third arm will often comprise a linkage having a series of joints,and a brake system coupled to the joints to releasably inhibitarticulation of the linkage. The third arm linkage will preferably alsohave a repositionable configuration allowing manual articulation of thearm, and at least some of the arms will often remain stationary and/orbe repositionable in response to a signal. In addition to comprising arobotic manipulator arm having a driven configurations, the third armmay alternatively comprise a simple passive linkage with a brake systembut without actuators.

Preferably, the surgical system will include four or more roboticmanipulator arms. One of the additional arms may support an imagecapture device of the imaging system. To allow the operator toselectively manipulate all of these surgical instruments, including theimage capture device, the processor will have an operation mode in whichthe first arm moves its associated surgical instrument at the surgicalsite in response to movement to the first input device, while the secondarm moves its associated surgical instrument in response to movement ofthe second input device. In response to an arm selection signal from atleast one arm selector input coupled to the processor, the processor canselectively change operating modes by decoupling the first arm from thefirst input device, and instead operatively couple the first inputdevice with the second arm, the third arm, or the fourth arm. Theprocessor will maintain some (or ideally all) of the decoupled arms inthe stationary configuration under some circumstances, although adecoupled arm could be controlled to move in a repetitive or automatedmanner until re-coupled to the surgeon's input device. An example ofsuch automated motion of a decoupled robotic arm includes motiontracking of a beating heart.

The robotic surgical system will often include an assistant inputdevice. The processor can selectively associate one or more of the armswith the assistant input device, or with an input device of the surgeon,so that the one arm moves in response to movement of the selected inputdevice. Hence, the processor can “hand-off” control of at least one arm(and its associated surgical instrument) between surgeon and assistantinput stations. This may be useful when the assistant is removing andreplacing the surgical instrument from the arm or when the assistant ata second console is to perform a part of the surgical procedure such as“closing” a portion of the surgical site. This also allows the surgeonto selectively assign direct control over an instrument based on theskill required to use the instrument for a given task, thereby enhancingrobotic team effectiveness. The assistant may optionally be working atan assistant control station that can correlate direction of movement ofthe assistant input device with an image of the end effector shown in anassistant display. The “assistant” image might be either a differentimage from a second endoscope or a shared image from the primaryconsole's endoscope, thereby enabling the surgeon and assistant to viewthe same image of the surgical site and manipulate each of theirassigned and coupled instruments to cooperate in performing a surgicalprocedure. Alternatively, a simple video monitor and assistant inputdevice (such as a handle within the sterile field) may be provided forthe assistant, particularly for a patient-side assistant performing toolswaps, intermittent irrigation, or the like. In other embodiments, thesurgeon's ability to select from among three or more surgicalinstruments, and to selectively associate each instrument with each ofthe input devices, will reduce and/or eliminate the need for surgicalassistants. Decreasing the use of surgical assistants (and the time tocontinually direct and oversee the assistant's movements) cansignificantly decrease the time and expense of a surgical procedure.

Typically, for a system having four arms, for example, three surgicalinstruments and the image capture device will be supported by fourmanipulators each comprising an endoscopic instrument having an elongateshaft with a proximal end adjacent the manipulator and a distal endinsertable into an internal surgical site within a patient body througha minimally invasive surgical aperture. Preferably, the manipulatorswill support the surgical instrument so that the shafts extend radiallyoutwardly from a pattern of apertures (typically incisions) in a “spokedwheel”-type arrangement. In an exemplary arrangement for performingcardiac surgery, the four shafts will be sufficiently long to enterapertures on the right side of the patient body, and to extend towardthe left side of the patient body for treating the heart. During atleast a portion of the procedure, top and bottom apertures of thepattern may accommodate first and second endoscopes to provideflexibility in the field of view, particularly for the separate steps ofharvesting a suitable supply artery and forming of the anastomosisduring coronary artery bypass grafting (CABG). Additional ports cansupport additional surgical tools. When performing a CABG procedurewithout cardioplegia on a beating heart, at least one of the aperturesof the pattern will often accommodate a tissue stabilizer during atleast a portion of the procedure.

In another aspect, the invention provides a robotic surgical systemcomprising a plurality of manipulator arms and a plurality of surgicalinstruments, each instrument mounted to an associated arm. A mastercontroller station has a master display for viewing by an operator, afirst input device for manipulation by a first hand of the operator, anda second input device for manipulation by a second hand of the operator.An assistant input device for manipulation by a hand of an assistant isalso provided. Preferably, a processor selectively operatively couplesthe controllers to the arms to effect movement to the surgicalinstruments in response to movement of the input devices.

In a method aspect, the invention provides a robotic surgical methodcomprising robotically moving first and second surgical instruments at asurgical site with first and second robotic manipulator arms bymanipulating first and second input devices with first and second handsof the operator, respectively. The first input device is selectivelyassociatable with a third manipulator arm, so that a third surgicalinstrument can be robotically moved at the surgical site with the thirdmanipulator arm by manipulating the first input device with the firsthand of the operator.

In another method aspect, the invention provides a robotic surgicalmethod comprising robotically moving first and second surgicalinstruments at a surgical site with first and second robotic manipulatorarms by manipulating first and second input devices with first andsecond hands of the operator, respectively. A third surgical instrumentis positioned at the surgical site by articulating a linkage of a thirdmanipulator. Movement of the positioned third surgical instrument isimpeded at the surgical site by inhibiting movement of the thirdmanipulator.

In yet another method aspect, the invention provides a robotic surgicalmethod comprising robotically positioning a surgical instrument at asurgical site with a manipulator arm by manipulating a first inputdevice with a hand of a first operator. The manipulator arm isselectively associated with a second input device, and the surgicalinstrument is robotically moved at the surgical site with themanipulator arm by manipulating the second input device with a hand of asecond operator.

In a still further method aspect, the invention provides a roboticsurgical method comprising showing, on a display, a first view of asurgical site from a first image capture device. A surgical instrumentis robotically removed at the surgical site with a manipulator arm bymanipulating an input device with a hand of an operator while theoperator views the first view of the surgical site on the display. Asecond image capture device is selectively associated with the display,and the surgical instrument is robotically manipulated at the surgicalsite with the arm by manipulating the input device while the operatorviews a second view of the surgical site from the second image capturedevice on the display.

In yet another method aspect, the invention provides a robotic coronaryartery bypass grafting method comprising introducing an image capturedevice into a chest cavity of a patient through an aperture disposedalong a right side of the patient. An image of a surgical site adjacentthe heart is displayed from the image capture device to an operator. Asurgical procedure is performed on the heart by moving a surgicalinstrument at the surgical site with at least one robotic manipulatorarm while the surgical instrument extends through another aperturedisposed along the right side of the patient. Preferably, tissuemanipulation during the surgical procedure will primarily be performedby surgical tools extending through a pattern of apertures along theright side of the patient.

In another aspect, first and second robotic manipulators are controlledby a first operator with first and second controllers, and third andfourth manipulators are controlled by a second operator with third andfourth controllers. Both operators view an image of the operating sitecaptured by a single image capture device, and both cooperate to performa surgical procedure. Each operator may have a separate dedicatedviewing station for his use. Such cooperation may include helping eachother perform the procedure, passing objects back and forth betweenmanipulators during the procedure, and passing control of various of themanipulator arms. Both sets of manipulators may share the same referencepoint so that control may be transferred without loss of context.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described, by way of example, and withreference to the accompanying diagrammatic drawings, in which:

FIG. 1 is a plan view of a telesurgical system and method for performinga robotic minimally invasive surgical procedure;

FIG. 2 shows a three-dimensional view of a control station of atelesurgical system in accordance with the invention;

FIGS. 3A-C show three-dimensional views of an input device including anarticulated arm and wrist to be mounted on the arm for use in the mastercontrol station of FIG. 2.

FIG. 4 shows a three-dimensional view of a cart of the telesurgicalsystem in accordance with the invention, the cart carrying threerobotically controlled manipulator arms, the movement of the arms beingremotely controllable from the control station shown in FIG. 2;

FIGS. 5 and 5A show a side view and a perspective view, respectively, ofa robotic arm and surgical instrument assembly in accordance with theinvention;

FIG. 6 shows a three-dimensional view of a surgical instrument of theinvention;

FIG. 7 shows, at an enlarged scale, a wrist member and end effector ofthe surgical instrument shown in FIG. 6, the wrist member and endeffector being movably mounted on a working end of a shaft of thesurgical instrument;

FIGS. 8A-C illustrate alternative end effectors having surfaces forstabilizing and/or retracting tissue.

FIGS. 9A-E illustrate another cart supporting a fourth roboticmanipulator aim in the telesurgical system of FIG. 1, and a bracket formounting a tool on the manipulator arm.

FIG. 10 shows a schematic three-dimensional drawing indicating thepositions of the end effectors relative to a viewing end of an endoscopeand the corresponding positions of master control input devices relativeto the eyes of an operator, typically a surgeon;

FIG. 11 shows a block diagram indicating one embodiment of a controlsystem of the telesurgical system of the invention;

FIGS. 11A-D schematically illustrate block diagrams and datatransmission time lines of an exemplary controller for flexibly couplingmaster/slave pairs;

FIG. 12 shows a block diagram indicating the steps involved in movingthe position of one of the master controls relative to its associatedend effector;

FIG. 13 shows a control diagram which indicates control steps involvedwhen the master control is moved relative to its associated end effectoras indicated in the block diagram of FIG. 12;

FIG. 14 shows a block diagram indicating the steps involved in movingthe position of one of the end effectors relative to its associatedmaster control;

FIG. 15 shows a control diagram which indicates control steps involvedwhen the end effector is moved relative to its associated master controlas indicated in FIG. 14;

FIG. 16 shows a block diagram indicating the steps involved in movingthe position of a viewing end of an endoscope of the minimally invasivetelesurgical system relative to the end effectors;

FIG. 17 shows a control diagram which indicates control steps involvedwhen the end of the endoscope is moved relative to the end effectors asindicated in FIG. 16;

FIG. 18 shows a simplified block diagram indicating the steps involvedin realigning a master control device relative to its associated endeffector;

FIG. 19 shows a block diagram indicating the steps involved inre-connecting a control loop between a master control device and itsassociated end effector;

FIG. 19A shows a block diagram indicating the steps involved in smoothlyrecoupling an input device with an end effector so as to avoidinadvertent sudden movements;

FIG. 20 shows a schematic diagram indicating an operator of theminimally invasive telesurgical system of the invention at the controlstation shown in FIG. 2;

FIG. 21 shows a schematic diagram of an image captured by an endo scopeof the minimally invasive telesurgical system of the invention asdisplayed on a viewer of the system;

FIG. 21A shows a schematic diagram of another image captured by theendoscope of the minimally invasive telesurgical system of the inventionas displayed on the viewer of the system;

FIG. 22 shows another image captured by the endoscope of the minimallyinvasive telesurgical system of the invention as displayed on the viewerof the system;

FIG. 22A shows a reference plane indicating a region in dashed lineswhich corresponds to an area where an automated determination of whichof two master control devices of the system is to be associated withwhich of two slaves of the system is not desired;

FIG. 22B shows a block diagram indicating steps involved in determiningan association between which of the two master control devices is to beassociated with which of the two slaves of the system; and

FIG. 22C shows a block diagram indicating steps involved when theassociation of one of the master control devices with a slave is to beswitched or swapped with the association of another master controldevice and slave;

FIGS. 23A and 23B illustrate a system and method for performing coronaryartery bypass grafting on a beating heart by selectively associatingrobotic surgical instruments with master input control devices;

FIG. 24 shows a block diagram indicating steps involved in a method forhanding-off control of robotic tools between a surgeon and an assistant;

FIG. 25A is a cross section, looking towards the head, through a chestof a patient body, illustrating an endoscopic coronary artery bypassgraft procedure in which the heart is treated by robotic toolsintroduced via a pattern of apertures on the right side of the patient;and

FIG. 25B illustrates an exemplary pattern of four apertures for fourrobotic endoscopic instruments as used in the procedure of FIG. 25A.

FIGS. 26 and 27 schematically illustrate alternative robotictelesurgical systems.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

This application is related to the following patents and patentapplications, the full disclosures of which are incorporated herein byreference: PCT International Application No. PCT/US98/19508, entitled“Robotic Apparatus”, filed on Sep. 18, 1998 (Attorney Docket No.17516-005510PC), U.S. Patent Application Ser. No. 60/111,713, entitled“Surgical Robotic Tools, Data Architecture, and Use” (Attorney DocketNo. 17516-003200), filed on Dec. 8, 1998; U.S. Patent Application Ser.No. 60/111,711, entitled “Image Shifting for a Telerobotic System”(Attorney Docket No. 17516-002700), filed on Dec. 8, 1998; U.S. PatentApplication Ser. No. 60/111,714, entitled “Stereo Viewer System for Usein Telerobotic System” (Attorney Docket No. 17516-001500), filed on Dec.8, 1998; U.S. Patent Application Ser. No. 60/111,710, entitled “MasterHaving Redundant Degrees of Freedom” (Attorney Docket No. 17516-001400),filed on Dec. 8, 1998, U.S. Patent Application No. 60/116,891, entitled“Dynamic Association of Master and Slave in a Minimally InvasiveTelesurgery System” (Attorney Docket No. 17516-004700), filed on Jan.22, 1999; and U.S. Pat. No. 5,808,665, entitled “Endoscopic SurgicalInstrument and Method for Use,” issued on Sep. 15, 1998.

As used herein, first and second objects (and/or their images) appear“substantially connected” if a direction of an incremental positionalmovement of the first object matches the direction of an incrementalpositional movement of the second object (often as seen in an image),regardless of scaling between the movements. Matching directions neednot be exactly equal, as the objects (or the object and the image) maybe perceived as being connected if the angular deviation between themovements remains less than about ten degrees, preferably being lessthan about five degrees. Similarly, objects and/or images may beperceived as being “substantially and orientationally connected” if theyare substantially connected and if the direction of an incrementalorientational movement of the first object is matched by the directionof an incremental orientational movement of the second object (often asseen in an image displayed near the first object), regardless of scalingbetween the movements.

Additional levels of connectedness may, but need not, be provided.“Magnitude connection” indicates substantial connection and that themagnitude of orientational and/or positional movements of the firstobject and second object (typically as seen in an image) are directlyrelated. The magnitudes need not be equal, so that it is possible toaccommodate scaling and/or warping within a magnitude connected roboticsystem. Orientational magnitude connection will imply substantial andorientational connection as well as related orientational movementmagnitudes, while substantial and magnitude connection means substantialconnection with positional magnitudes being related.

As used herein, a first object appears absolutely positionally connectedwith an image of a second object if the objects are substantiallyconnected and the position of the first object and the position of theimage of the second object appear at least to substantially match, i.e.,to be at the same location, during movement. A first object appearsabsolutely orientationally connected with an image of the second objectif they are substantially connected and the orientation of the firstobject and the second object at least substantially match duringmovement.

Referring now to FIG. 1, a robotic surgical network 10 includes a mastercontrol station 200 and a slave cart 300, along with any of severalother additional components to enhance the capabilities of the roboticdevices to perform complex surgical procedures. An operator O performs aminimally invasive surgical procedure at an internal surgical sitewithin patient P using minimally invasive surgical instruments 100.Operator O works at master control station 200. Operator O views adisplay provided by the workstation and manipulates left and right inputdevices. The telesurgical system moves surgical instruments mounted onrobotic arms of slave cart 300 in response to movement of the inputdevices. As will be described in detail below, a selectably designated“left” instrument is associated with the left input device in the lefthand of operator O, and a selectably designated “right” instrument isassociated with the right input device in the right hand of theoperator.

As described in more detail in co-pending U.S. patent application Ser.No. 09/373,678 entitled “Camera Referenced Control In A MinimallyInvasive Surgical Apparatus” and filed Aug. 13, 1999, (Attorney DocketNo. 17516-002110), the full disclosure of which incorporated herein byreference, a processor of master controller 200 will preferablycoordinate movement of the input devices with the movement of theirassociated instruments so that the images of the surgical tools 100, asdisplayed to the operator, appear at least substantially connected tothe input devices in the hands of the operator. Further levels ofconnection will also often be provided to enhance the operator'sdexterity and ease of use of surgical instruments 100.

Introducing some of the other components of network 10, an auxiliarycart 300A can support one or more additional surgical tools 100 for useduring the procedure. One tool is shown here for illustrative purposesonly. A first assistant A1 is seated at an assistant control station200A, the first assistant typically directing movements of one or moresurgical instruments not actively being manipulated by operator O viamaster control station 200. A second assistant A2 may be disposedadjacent patient P to assist in swapping instruments 100 during thesurgical procedure. Auxiliary cart 300A may also include one or moreassistant input devices 12 (shown here as a simple joy stick) to allowsecond assistant A2 to selectively manipulate the one or more surgicalinstruments while viewing the internal surgical site via an assistantdisplay 14. Preferably, the first assistant A1 seated at console 200Aviews the same image as surgeon seated at console 200. Furtherpreferably, both the instruments of cart 300 and the “assistant”instruments of cart 300A are controlled according to the same camerareference point, such that both surgeon and assistant are able to be“immersed” into the image of the surgical field when manipulating any ofthe tools.

As will be described hereinbelow, master control station 200, assistantcontroller 200A, cart 300, auxiliary cart 300A, and assistant display 14(or subsets of these components) may allow complex surgeries to beperformed by selectively handing-off control of one or more robotic armsbetween operator O and one or more assistants. Alternatively, operator Omay actively control two surgical tools while a third remains at a fixedposition, for example, to stabilize and/or retract tissue, with theoperator selectively operating the retractor or stabilizer only atdesignated times. In still further alternatives, a surgeon and anassistant can cooperate to conduct an operation without either passingcontrol of instruments or being able to pass control of the instruments,with both instead manipulating his or her own instruments during thesurgery, as will be described below with reference to FIG. 26. AlthoughFIG. 1 depicts two surgeon consoles controlling the two cart structures,a preferred embodiment comprises only one console controlling four ormore arms on two carts. The scope may optionally be mounted on theauxiliary cart and three tissue manipulator arms may be mounted on themain cart. Generally, the use of robotic systems having four or morearms will facilitate complex robotic surgical procedures, includingprocedures that benefit from selectable endoscope viewing angles.Methods for using robotic network 10 will be described in more detailfollowing descriptions of the network components. While the networkcomponent connections are schematically illustrated in FIGS. 1, 26, and27, it should be understood that more complex interconnections betweenthe various network components may be provided.

Component Descriptions.

Referring to FIG. 2 of the drawings, the control station of a minimallyinvasive telesurgical system in accordance with the invention isgenerally indicated by reference numeral 200. The control station 200includes a viewer or display 202 where an image of a surgical site isdisplayed in use. A support 204 is provided on which an operator,typically a surgeon, can rest his or her forearms while gripping twomaster controls (FIGS. 3A and 3B), one in each hand. The master controlsare positioned in a space 206 inwardly beyond the support 204. Whenusing control station 200, the surgeon typically sits in a chair infront of the control station 200, positions her eyes in front of theviewer 202, and grips the master controls, one in each hand, whileresting her forearms on the support 204.

An example of one of the master control input devices is shown in FIGS.3A-C, and is generally indicated by reference numeral 210. The mastercontrol device generally comprises an articulate positioning arm 210Asupporting orientational gimbals 210B. Gimbals 210B (shown most clearlyin FIG. 3B) have a plurality of members or links 212 connected togetherby joints 214, typically by rotational joints. The surgeon grips themaster control 210 by positioning his or her thumb and index finger overa grip actuation handle, here in the form of a grip handle or pincherformation 216. The surgeon's thumb and index finger are typically heldon the pincher formation by straps (not shown) threaded through slots218. To move the orientation of the end effector, the surgeon simplymoves the pincher formation 216 to the desired end effector orientationrelative to the image viewed at the viewer 202, and the end effectororientation is caused to follow the orientation of the pincherformation. Appropriately positioned positional sensors, e.g., encoders,or potentiometers, or the like, are coupled to each joint of gimbals210B, so as to enable joint positions of the master control to bedetermined as also described in greater detail herein below.

Gimbals 210B are similarly repositioned by movement of pincher formation216, and this positional movement is generally sensed by articulation ofinput arm 210A as shown in FIG. 3A. Reference numerals 1-3 indicateorientational degrees of freedom of gimbals 210B, while numeral 4 inFIGS. 3A and 3B indicates the joint with which the master control andthe articulated arm are connected together. When connected together, themaster control 210 can also displace angularly about axis 4.

The articulated arm 210A includes a plurality of links 220 connectedtogether at joints 222. Articulated arm 210A has appropriatelypositioned electric motors to provide for feedback as described ingreater detail below. Furthermore, appropriately positioned positionalsensors, e.g., encoders, or potentiometers, or the like, are positionedon the joints 222 so as to enable joint positions of the master controlto be determined as further described herein below. Axes A, B, and Cindicate the positional degrees of freedom of articulated arm 210A. Ingeneral, movement about joints of the master control 210B primarilyaccommodates and senses orientational movement of the end effector, andmovement about the joints of arm 210A primarily accommodates and sensestranslational movement of the end effector. The master control 210 isdescribed in greater detail in U.S. Provisional Patent Application Ser.No. 60/111,710, and in U.S. patent application Ser. No. 09/398,507,filed concurrently herewith (Attorney Docket No. 17516-001410), the fulldisclosures of which are incorporated herein by reference.

As described more fully in co-pending U.S. patent application Ser. No.09/373,678, the full disclosure of which is incorporated herein byreference, the orientation of the viewer relative to the master controlinput devices will generally be compared with the position andorientation of the end effectors relative to a field of view of theimage capture device. The relative locations of the input devices can bederived from knowledge regarding the input device linkage jointconfigurations (as sensed by the joint sensors), the construction anddesign of the master controller structure, and in some cases,calibration measurements taken from a specific master control consolesystem after fabrication. Such calibration measurements may be stored ina non-volatile memory of the console.

In FIG. 4 of the drawings, the cart 300 is adapted to be positionedclose to a surgical platform in preparation for surgery, and can then becaused to remain stationary until a surgical procedure has beencompleted. The cart 300 typically has wheels or castors to render itmobile. The control station 200 may optionally be positioned remote fromthe cart 300, but will often be in or adjacent the operating room. Thecart 300 carries three robotic arm assemblies. One of the robotic armassemblies, indicated by reference numeral 302, is arranged to hold animage capturing device 304, e.g., an endoscope, or the like. Each of thetwo other arm assemblies 310 is arranged to carry a surgical instrument100. The robotic arms are supported by positioning linkages 395, whichcan be manually positioned and then locked in place before (orre-positioned during) the procedure.

The positioning linkages or “set-up joints” are described in ProvisionalApplication Ser. No. 60/095,303, the full disclosure of which isincorporated herein by reference. Preferably, the set-up joints includejoint sensors which transmit signals to the processor indicating theposition of the remote center of rotation. It should be noted that themanipulator arm assemblies need not be supported by a single cart. Someor all of the manipulators may be mounted to a wall or ceiling of anoperating room, separate carts, or the like. Regardless of the specificmanipulator structures or their mounting arrangement, it is generallypreferable to provide information to the processor regarding thelocation of insertion/pivot points of the surgical instruments into thepatient body. The set-up joint linkages need not have joint drivesystems but will often include a joint brake system, as they will oftenhold the manipulators in a fixed position during some or all of asurgical procedure.

The endoscope 304 has a viewing end 306 at a remote end of an elongateshaft. The elongate shaft of endoscope 304 permits it to be insertedinto an internal surgical site of a patient's body. The endoscope 304 isoperatively connected to the viewer 202 to display an image captured atits viewing end 306 on the viewer 202.

Each robotic arm 302, 310 can be operatively connected to one or more ofthe master controls 210 so that the movement of instruments mounted onthe robotic arms is controlled by manipulation of the master controls.The instruments 100 carried on the robotic arm assemblies 310 have endeffectors, generally indicated at 102, which are mounted on wristmembers, the wrists in turn being pivotally mounted on distal ends ofelongate shafts of the instruments. It will be appreciated that theinstruments have elongate shafts to permit them to be inserted into aninternal surgical site of a patient's body. Movement of the endeffectors relative to the ends of the shafts of the instruments is alsocontrolled by the master controls.

In FIGS. 5 and 5A of the drawings, one of the robotic manipulator armassemblies 310 is shown in greater detail. Assembly 310 includes anarticulated robotic arm 312, and a surgical instrument, schematicallyand generally indicated by reference numeral 100, mounted thereon.

FIG. 6 indicates the general appearance of the surgical instrument 100in greater detail. The surgical instrument 100 includes an elongateshaft 104. The wrist-like mechanism, generally indicated by referencenumeral 106, is located at a working end of the shaft 104. A housing108, arranged to releasably couple the instrument 100 to the robotic arm312, is located at an opposed end of the shaft 104. In FIG. 6, and whenthe instrument 100 is coupled or mounted on the robotic arm 312, theshaft 104 extends along an axis indicated at 109.

Referring again to FIGS. 5 and 5A, the instrument 100 is typicallyreleasably mounted on a carriage 314, which is driven to translate alonga linear guide formation 316 of the arm 312 in the direction of arrowsP. The robotic arm 312 is typically mounted on a base by means of abracket or mounting plate 317, which is affixed to the passively movableset-up joints 395. Set-up joints 395 are held in a fixed configurationduring manipulation of tissue by a set-up joint brake system. The basemay be defined by the mobile cart or trolley 300, which is retained in astationary position during a surgical procedure.

The robotic arm 312 includes a cradle, generally indicated at 318, anupper arm portion 320, a forearm portion 322 and the guide formation316. The cradle 318 is pivotally mounted on the plate 317 in gimbaledfashion to permit rocking movement of the cradle in the direction ofarrows 326 as shown in FIG. 5, about a pivot axis 328. The upper armportion 320 includes link members 330, 332 and the forearm portion 322includes link members 334, 336. The link members 330, 332 are pivotallymounted on the cradle 318 and are pivotally connected to the linkmembers 334, 336. By use of this linkage, irrespective of the movementof the robotic arm 312, a pivot center 349 remains in the same positionrelative to plate 317 with which the arm 312 is mounted. In use, thepivot center 349 is positioned at an aperture or a port of entry into apatient's body when an internal surgical procedure is to be performed.

While this “remote” center of motion-type arrangement for roboticmanipulation is described in connection with the preferred embodimentsof this invention, the scope of the inventions disclosed herein is notso limited, encompassing other types of arrangements such as manipulatorarms having passive or natural centers of motion at the point ofinsertion into a patient body.

The robotic arm 312 provides three degrees of freedom of movement to thesurgical instrument 100 when mounted thereon. These degrees of freedomof movement are firstly the gimbaled motion indicated by arrows 326,pivoting movement as indicated by arrows 327 and the linear displacementin the direction of arrows P. These three degrees of freedom of movementare primarily coupled to translational degrees of movement of the endeffector, although some rotational coupling may be present. Movement ofthe arm as indicated by arrows 326, 327 and P is controlled byappropriately positioned electrical motors which respond to inputs froman associated master control to drive the arm 312 to a required positionas dictated by movement of the master control. Appropriately positionedsensors, e.g., potentiometers, or the like, are provided on the arm todetermine joint positions as described in greater detail herein below.

Referring now to FIG. 7 of the drawings, the wrist mechanism 106 willnow be described in greater detail. In FIG. 7, the working end of theshaft 104 is indicated at 110. The wrist mechanism 106 includes a wristmember 112. One end portion of the wrist member 112 is pivotally mountedin a clevis, generally indicated at 117, on the end 110 of the shaft 104by means of a pivotal connection 114. The wrist member 112 can pivot inthe direction of arrows 156 about the pivotal connection 114.

An end effector, generally indicated by reference numeral 102, ispivotally mounted on an opposed end of the wrist member 127. The endeffector 102 is in the form of, e.g., a clip applier for anchoring clipsduring a surgical procedure. Accordingly, the end effector 102 has twoelements 102.1, 102.2 together defining a jaw-like arrangement. It willbe appreciated that the end effector can be in the form of any surgicaltool having two members which pivot about a common pivotal axis, such asscissors, pliers for use as needle drivers, or the like. Instead, it caninclude a single working member, e.g., a scalpel, cautery electrode, orthe like. Alternative non-articulated tools may also be used, includingtools for aspiration and/or irrigation, endoscopes, or the like. When atool other than a clip applier is required during the surgicalprocedure, the tool 100 is simply removed from its associated arm andreplaced with an instrument bearing the required end effector, e.g., ascissors, or pliers, or the like.

The end effector 102 is pivotally mounted in a clevis, generallyindicated by reference numeral 119, on an opposed end of the wristmember 112, by means of a pivotal connection 160. Elements 102.1, 102.2are angularly displaceable about the pivotal connection 160 toward andaway from each other as indicated by arrows 162, 163. It will further beappreciated that the elements 102.1, 102.2 can be displaced angularlyabout the pivotal connection 160 to change the orientation of the endeffector 102 as a whole, relative to the wrist member 112. Thus, eachelement 102.1, 102.2 is angularly displaceable about the pivotalconnection 160 independently of the other, so that the end effector 102,as a whole, is angularly displaceable about the pivotal connection 160as indicated in dashed lines in FIG. 7. Furthermore, the shaft 104 isrotatably mounted on the housing 108 for rotation as indicated by thearrows 159. Thus, the end effector 102 has three degrees of freedom ofmovement relative to the arm 112 in addition to actuation of the endeffector, preferably namely, rotation about the axis 109 as indicated byarrows 159, angular displacement as a whole about the pivot 160 andangular displacement about the pivot 114 as indicated by arrows 156.Other wrist structures and combinations of joints also fall within thescope of the present inventions, however. For example, while thisarrangement and these resulting degrees of freedom of movement arepreferred, a wrist having fewer degrees of freedom of movement, such asa single distal articulating joint, or a wrist having othersingularities, may also be used, as desired.

The three degrees of freedom of movement of instrument 100 are primarilycoupled to orientational degrees of freedom of movement of the endeffector. This is somewhat a simplification, as movement about thesethree axes will result in some change in position of the end effector.Similarly, movement about the above-described translational axes maycause some changes in orientation. It will be appreciated thatorientational movement of the end effector, like translational movement,is controlled by appropriately positioned electrical motors whichrespond to inputs from the associated master control to drive the endeffector 102 to a desired position as dictated by movement of the mastercontrol. Furthermore, appropriately positioned sensors, e.g., encoders,or potentiometers, or the like, are provided to determine jointpositions as described in greater detail herein below. In thisspecification the actuation or movement of the end effectors relative toeach other in the directions of arrows 62, 63 is not regarded as aseparate degree of freedom of movement.

Tissue stabilizer end effectors 120 a, b, and c, referred to generallyas tissue stabilizers 120, are illustrated in FIGS. 8A-C. Tissuestabilizers 120 may have one or two end effector elements 122 thatpreferably are pivotally attached to the distal end of the shaft orwrist of a surgical instrument and are moveable with respect to oneanother, and that preferably comprise tissue-engaging surfaces 124. Thetissue-engaging surfaces optionally include protrusions, ridges, vacuumports, or other surfaces adapted so as to inhibit movement between theengaged tissue and the stabilizer, either through pressure applied tothe engaged tissue or vacuum applied to draw the tissue into an at leastpartially stabilized position, or a combination of both pressure andvacuum. The ideal tissue engaging surface will constrain and/or reducemotion of the engaged tissue in the two lateral (sometimes referred toas the X and Y) axes, along the tissue-engaging surface, and thestabilizer configuration and engagement with the tissue will at leastpartially decrease motion normal to the surface. Other configurationsfor traditional stabilizers are known to those of skill in the art, suchas the Octopus II of Medtronic, Inc. and various HeartPort, Inc. andCardioThoracic Systems stabilizers having multipronged and doughnutconfigurations. These manners of contacting tissue allow stabilizers 120to firmly engage a moving tissue such as a beating heart of a patientand reduce movement of the tissue adjacent the stabilizer.

To facilitate performing a procedure on the stabilized tissue, anopening 126 may be formed in an individual stabilizer element 122,and/or between independently moveable end effector elements. Asillustrated in FIG. 8B, stabilizer 120 b includes cooperating tissuegrasping surfaces 128 disposed between stabilizer end effector elements122. This allows the stabilizer to grasp tissues, providing a dualfunction robotic stabilizer/grasper tool. Stabilizer 120 b may be used,for example, as a grasper while harvesting and/or preparing an internalmammary artery (IMA) for a coronary artery bypass graft (CABG)procedure, and/or to hold the IMA during formation of the anastomosis onthe stabilized beating heart.

In general, tissue stabilizers 120 will have a sufficiently smallprofile, when aligned with shaft 104 of instrument 100, to allow thestabilizer to advance axially through a cannula. Similar (or modified)end effectors having high friction tissue-engaging surfaces may be usedas retractors to hold tissue clear of a surgeon's line of sight during aprocedure.

Referring now to FIG. 8C, and generally for the robotic endoscopicstabilizers disclosed herein, each stabilizer may comprise an irrigationport 125, the port preferably in fluid communication with a lumenintegrated into the shaft of the stabilizer tool. While an irrigationand/or aspiration capability is particularly beneficial whenincorporated into a stabilizer, such capabilities may also beincorporated into the shaft of any robotic surgical tool, as desired.The port system, comprising a lumen preferably situated inside the shaftof the stabilizer and extending out of an aperture or port in the distalportion of the shaft, may be used to perform a number of tasks during asurgical procedure (e.g., a beating heart procedure) in whichstabilization of tissue is desired. Those tasks may include removingundesired fluid from the surgical site (either through suction tooutside the patient's body), blowing the fluid into some other portionof the surgical site, and/or delivering fluid (such as spray humidifiedcarbon dioxide) to clear the surgical site of material (such as bodyfluids which might otherwise interfere with the surgeon's view).Preferably, at least the distal portion of the port system is flexibleto permit bending. The exemplary port structure will be malleable orplastically deformable enough that it will hold its position whenrepositioned.

To take advantage of the irrigation aspect of this multi-functionalstabilizer, the stabilizer is inserted with the distal external portionof the irrigation device preferably flush with the shaft of thestabilizer. After the stabilizer has reached the surgical site, theoperator may reposition the irrigation port distal end with one of theother surgical manipulators by grasping the port structure and moving itto a desired location and/or orientation relative to shaft 104, wrist106, or end effector element 122 (depending on the structure to whichthe port is mounted). The device may remain in that location for theduration of the surgery, or may be moved around as desired. In additionto simply being moveable at the surgical site, the device also may beextendable from/retractable into the stabilizer shaft, so that thedistal end can be moved towards or away from the surgical site itself,as desired.

An example of a preferred auxiliary cart 300A is seen in more detail inFIGS. 9A and B. Auxiliary cart 300A includes a simple linkage 350 withsliding joints 352 which can be releasably held in a fixed configurationby latches 354. Linkage 350 supports an auxiliary remote centermanipulator arm 302A having a structure similar to arm 302 used tosupport the endoscope on cart 300. (See FIG. 4.) The linkage structureof auxiliary arm 302A is described more fully in co-pending U.S. PatentApplication Ser. No. 60/112,990, filed on Dec. 16, 1998, the fulldisclosure of which is incorporated herein by reference. Generally,auxiliary arm 302A effectively includes a parallel linkage mechanismproviding a remote center of spherical rotation 349 at a fixed locationrelative to base 317, similar to that described above with reference toarm 312 in FIG. 5. Although this arm is described as preferably being ofdifferent structure that other instrument manipulator arms alsodescribed herein, it should be understood that a other manipulator armcan also be used either to support an endoscope or to serve as thefourth arm on the auxiliary cart 300A.

Sliding joints 352 and wheels 356 (which can also be releasably lockedin a fixed configuration by latches 354) allow remote center or fulcrum349 to be positioned at an insertion point into a patient body usingtranslational movement along X, Y, and Z axes. Auxiliary arm 302A mayoptionally be actively driven so as to translationally position a shaftof a surgical instrument within a patient body. Alternatively, auxiliaryarm 302A may be used as a passive manipulator arm. Auxiliary arm 302A(like all manipulator arms of the robotic network) preferably includes arepositioning configuration input device or button 358, ideally disposedon a manual positioning handle 360. When repositioning button 358 isdepressed, the joints of auxiliary arm 302A move freely so as to pivotthe arm about fulcrum 349 manually. Once actuator 358 is released,auxiliary arm 302A remains in a substantially fixed configuration. Thearm will resist movement until repositioning button 358 is again helddown, or until the arm receives an actuation signal from an associatedmaster control input device. Hence, auxiliary cart 300A may be used tosupport a surgical instrument such as an endoscope, a stabilizer, aretractor, or the like, even if not actively driven under direction ofan input device.

Manual repositioning of the supported surgical instrument will generallybe performed by an assistant under the direction of a surgeon in chargeof the surgical procedure. Typically, even when the set-up joints 395,cart linkages 350, arms 302, 312, and/or other structures of the roboticsystem support the end effectors in a fixed configuration, the brake ormotor drive systems inhibiting movement of the instruments can be safelyoverridden using manual force without damaging the robotic system. Thisallows repositioning and/or removal the instruments if a failure occurs.Preferably, the override force will be sufficient to inhibit inadvertentmovement from accidental bumping, interference between manipulators, andthe like.

Auxiliary arm 302A and arm 302 used to support endoscope 304 need notnecessarily include a drive system for articulating a wrist and/or endeffectors within the patient body, unless, e.g., a wrist is to be usedin connection with a stabilizer to improve positioning of the particulartissue to be stabilized. When auxiliary cart 300A is to be used toactively drive an articulated tool under the direction of an operator Oor assistant via a processor, arm 302 may optionally be replaced by arm312. Alternatively, where the auxiliary cart is to be used as a passivestructure to hold an articulated surgical instrument at a fixed positionand configuration within a patient body, a manual tool articulationbracket 370 may be used to mount the tool 100 to auxiliary arm 302A. Themanual tool bracket 370 is illustrated in FIGS. 9C-9E.

As can be understood with references to FIGS. 9C and 9D, bracket 370comprises a plate 372 with sidewalls which fittingly receive housing 108of tool 100. Discs 374 have drive surfaces which drivingly engage thedrive system of tool 100 so as to rotate shaft 104 about its axis,articulate the end effector about the wrist, and move the first andsecond end effector elements, as described above.

As seen most clearly in FIG. 9E, the rotational position of discs 374can be changed by manually rotating adjustment knobs 376, which arerotationally coupled to the discs. Once the instrument 100 is in thedesired configuration, lock nuts 378 may be tightened against washers379 to rotationally affix knobs 376 and discs 374. In the exemplaryembodiment, bracket 372 comprises a polymer, while knobs 376 and nuts378 may be polymeric and/or metallic. Washer 379 may comprise a lowfriction polymer, ideally comprising a PTFE such as Teflon™, or thelike. While the disclosure herein shows a preferred embodiment formanual manipulation of a stabilizer by a surgical assistant, it shouldbe apparent that the stabilizer might just as easily be controlled froma remote robotic control console, from which the operator wouldmanipulate the stabilizer and any associated wrist in the same way asother instruments are controlled, as herein described.

Telesurgical Methods and Component Interactions.

In use, the surgeon views the surgical site through the viewer 202. Theend effector 102 carried on each arm 312, 302, 302A is caused to performmovements and actions in response to movement and action inputs of itsassociated master control. It will be appreciated that during a surgicalprocedure images of the end effectors are captured by the endoscopetogether with the surgical site and are displayed on the viewer so thatthe surgeon sees the movements and actions of the end effectors as he orshe controls such movements and actions by means of the master controldevices. The relationship between the end effectors at the surgical siterelative to the endoscope tip as viewed through the viewer and theposition of the master controls in the hands of the surgeon relative tothe surgeon's eyes at the viewer provides an appearance of at least asubstantial connection between the master controls and the surgicalinstrument for the surgeon.

To provide the desired substantial connection between the end effectorimages and the master controller input devices, the processor of mastercontrol station 200 and/or assistant control station 200A will generallymap the internal surgical worksite viewed by the endoscope onto themaster controller work space in which the operator and/or assistantmoves his or her hands. The position of the arms holding the surgicaltools relative to the arm holding the endoscope in use may be used toderive the desired coordinate transformations so as to provide thedesired level of substantial connectedness, as more fully explained inco-pending U.S. Patent Provisional Application Ser. No. 60/128,160,previously incorporated herein by reference.

Where a tool is to be viewed through an endoscope, and the tool andendoscope are supported by independent support structures (for example,when viewing a tool supported by arm 312 within the internal surgicalsite via an endoscope supported by auxiliary cart 300A) it isparticularly beneficial to have a known orientation between the twoindependent support structures to allow the desired transformations tobe derived. This may be provided, for example, by ensuring that the basestructure of cart 300 is accurately parallel to the base structure ofauxiliary cart 300A. As positional transformations and modifications arerelatively straightforward when orientations are accurately aligned,this allows a processor to provide substantial connection despite theseparately mounted robotic network components.

The operation of telesurgical robotic network 10 will first be explainedwith reference to interaction between master control station 200 andcart 300. Many of the aspects of this interaction appear in theinteractions among the remaining network components.

Master-Slave Controller.

In FIG. 10, the Cartesian space coordinate system is indicated generallyby reference numeral 902. The origin of the system is indicated at 904.The system 902 is shown at a position removed from the endoscope 304. Inthe minimally invasive telesurgical system of the invention, and forpurposes of identifying positions in Cartesian space, the origin 904 isconveniently positioned at the viewing end 306. One of the axes, in thiscase the Z-Z axis, is coincident with the viewing axis 307 of theendoscope. Accordingly, the X-X and Y-Y axes extend outwardly indirections perpendicular to the viewing axis 307.

It will be appreciated that in the case of angular displacement of theendoscope to vary the orientation of the displayed image as describedabove, the reference plane defined by the X-X and Y-Y axis is angularlydisplaced together with the endoscope.

As mentioned earlier, when the surgical instruments are mounted on thearms 112, a fulcrum 349 or pivot point is defined for each arm assembly310. Furthermore, as also already mentioned, each fulcrum 349 ispositioned at a port of entry into the patient's body. Thus, movementsof the end effectors at the surgical site is caused by angulardisplacements about each fulcrum 349. As described above, the locationof the fulcrums may be sensed using joint sensors of the set-up joints,or using a variety of alternative position sensing systems.

When the remote center or fulcrum positions relative to the viewing end306 of the endoscope 304 are determined, the coordinates in the X-X andY-Y plane of the Cartesian coordinate system 902 are determined. It willbe appreciated that these (X,Y) coordinates of each fulcrum 349 can varydepending on the chosen entry ports to the surgical site. The locationof these entry ports can vary depending on the surgical procedure to beperformed. It will further be appreciated that the (X,Y) coordinates ofeach fulcrum 349 can readily be determined with reference to thecoordinate system 902 by means of the position sensors at the variouspivot points on each robotic arm 112 since the endoscope 304 and thearms 310 are mounted on the same cart 300. Naturally, the endoscope arm302 is also provided with appropriately positioned positional sensors.Thus, to determine the (X,Y) coordinates of each fulcrum 349, relativeto the coordinate system 902, the position of the coordinate system 902can be determined relative to any arbitrary point in space by means ofthe positional sensors on the endoscope arm 302 and the positions ofeach fulcrum relative to the same arbitrary point can readily bedetermined by means of the positional sensors on each robotic arm 112.The positions of each fulcrum 349 relative to the coordinate system 902can then be determined by means of routine calculation.

With reference to FIG. 11, a control system defining a control loopwhich links master control inputs to end effector outputs, and viceversa for feedback, is schematically indicated by reference numeral 400.Master control inputs and corresponding end effector outputs areindicated by arrows AB and end effector inputs and corresponding mastercontrol outputs in the case of feedback is indicated by arrows BA.

In this specification, for the sake of clarity, positions sensed by theencoders on the master which relate to joint positions are referred toas “joint space” positions. Similarly, for the sensors on the joints ofthe robotic arm and the wrist mechanism, positions determined by thesesensors are also referred to as “joint space” positions. The robotic armand wrist mechanism will be referred to as the slave in the descriptionwhich follows. Furthermore, references to positions and positionedsignals may include orientation, location, and/or their associatedsignals. Similarly, forces and force signals may generally include bothforce and torque in their associated signals.

For ease of explanation, the system 400 will be described from aninitial condition in which the master is at an initial position and theslave is at a corresponding initial position. However, in use, the slavetracks master position in a continuous manner.

Referring to the control system 400, the master is moved from theinitial position to a new position corresponding to a desired positionof the end effector as viewed by the surgeon in the image displayed onthe viewer 202. Master control movements are input by a surgeon at 402,as indicated by arrow AB1 by applying a force to the master control at404 to cause the master control to move from its initial position to thenew position.

As the master is moved, signals e_(m) from the encoders on the master isinput to a master input controller at 406 as indicated by arrow AB2. Atthe master input controller 406, the signals e_(m) are converted to ajoint space position θ_(m) corresponding to the new position of themaster. The joint space position θ_(m) is then input to a masterkinematics converter 408 as indicated by arrow AB3. At 408 the jointposition θ_(n), is transformed into an equivalent Cartesian spaceposition x_(m). This is optionally performed by a kinematic algorithmincluding a Jacobian transformation matrix, inverse Jacobian (J-¹), orthe like. The equivalent Cartesian space position x_(m) is then input toa bilateral controller at 410 as indicated by arrow AB4.

Position comparison and force calculation may, in general, be performedusing a forward kinematics algorithm which may include a Jacobianmatrix. Forward kinematics algorithm generally makes use of a referencelocation, which is typically selected as the location of the surgeon'seyes. Appropriate calibration or appropriately placed sensors on console200 can provide this reference information. Additionally, the forwardkinematics algorithm will generally make use of information concerningthe lengths and angular offsets of the linkage of the master inputdevice 210. More specifically, the Cartesian position x_(m) representsthe distance of the input handle from, and the orientation of the inputhandle relative to, the location of the surgeon's eyes. Hence, x_(m) isinput into bilateral controller 410 as indicated by AB4.

In a process similar to the calculations described above, the slavelocation is also generally observed using sensors of the slave system.In the exemplary embodiment, the encoder signal e, are read from theslave joint sensors at 416 as indicated by BA2, and are then convertedto joint space at step 414. As indicated by BA3, the joint spaceposition of the slave is also subjected to a forward kinematicsalgorithm at step 412. Here, the forward kinematics algorithm ispreferably provided with the referenced location of tip 306 of endoscope304. Additionally, through the use of sensors, design specifications,and/or appropriate calibration, this kinematics algorithm incorporatesinformation regarding the lengths, offsets, angles, etc., describing thelinkage structure of patient cart 300, set-up joints 395, and roboticmanipulator arms 310, so that the slave Cartesian position x_(s)transferred at BA4 is measured and/or defined relative to the endoscopetip.

At bilateral controller 410, the new position of the master x_(m) inCartesian space relative to the surgeon's eyes is compared with theinitial position x_(s) of the instrument tip in Cartesian space relativeto the camera tip. This relationship is depicted in FIG. 10 showing thetriangle connecting the surgeon's eye and the master controllers in thehands of the surgeon, as well as the triangle coupling camera tip 306and the end effectors of tools 104. Advantageously, the comparison ofthese relative relationships occurring in controller 410 can account fordifferences in scale between the master controller space in which theinput device is moved as compared with the surgical workspace in whichthe end effectors move. Similarly, the comparison may account forpossible fixed offsets, should the initial master and slave positionsnot correspond.

At 410, the new position x_(m) of the master in Cartesian space iscompared with the initial position of the slave, also in Cartesianspace. It will be appreciated that the positions of the master and slavein Cartesian space are continually updated in a memory. Thus, at 410,the initial position of the slave in Cartesian space is downloaded fromthe memory so as to compare it with the new position of the master inCartesian space. Thus, the initial position of the slave in Cartesianspace was derived from the joint space position of the slave when boththe master and the slave were at their initial positions. It willfurther be appreciated that, at 410, and where the position of themaster in Cartesian space conforms with a corresponding position of theslave in Cartesian space, no positional deviation results from thecomparison at 410. In such a case no signals are sent from 410 to causemovement of the slave or the master.

Since the master has moved to a new position, a comparison of itscorresponding position x_(m) in Cartesian space with the Cartesian spaceposition of the slave corresponding to its initial position, yields apositional deviation. From this positional deviation in Cartesian space,a force f_(s) in Cartesian space is computed at 410 which is necessaryto move the slave position in Cartesian space to a new positioncorresponding to the new position of the master x_(m) in Cartesianspace. This computation is typically performed using a proportionalintegral derivative (P.I.D.) controller. This force f_(s) is then inputto a slave kinematics converter 412 as indicated by arrow AB5.Equivalent joint torques τ_(s) are computed in the slave kinematicsmodule, typically using a Jacobian transpose method. This is optionallyperformed by a Jacobian Transpose (J^(T)) controller.

The torques τ_(s) are then input to a slave output converter at 414 asindicated by arrow AB6. At 414 currents is are computed. These currentsis are then forwarded to the electrical motors on the slave at 416 asindicated by arrow AB7. The slave is then caused to be driven to the newposition x_(e) which corresponds to the new position into which themaster has been moved.

The control steps involved in the control system 400 as explained aboveare typically carried out at about 1300 cycles per second or faster. Itwill be appreciated that although reference is made to an initialposition and new position of the master, these positions are typicallyincremental stages of a master control movement. Thus, the slave iscontinually tracking incremental new positions of the master.

The control system 400 makes provision for force feedback. Thus, shouldthe slave, typically the end effector, be subjected to an environmentalforce f_(e) at the surgical site, e.g., in the case where the endeffector pushes against tissue, or the like, such a force is fed back tothe master control. Accordingly, when the slave is tracking movement ofthe master as described above and the slave pushes against an object atthe surgical site resulting in an equal pushing force against the slave,which urges the slave to move to another position, similar steps asdescribed above take place.

The surgical environment is indicated at 418 in FIG. 11. In the casewhere an environmental force f_(e) is applied on the slave, such a forcef_(e) causes displacement of the end effector. This displacement issensed by the encoders on the slave 416 which generate signals e_(s).Such signals e_(s) are input to the slave input converter 414 asindicated by arrow BA2. At the slave input 414 a position θ_(s) in jointspace is determined resulting from the encoder signals e_(s). The jointspace position O is then input to the slave kinematics converter at 412and as indicated by arrow BA3. At 412 a Cartesian space position x_(s)corresponding to the joint space position θ_(s) is computed and input tothe bilateral controller at 410 as indicated by arrow BA4. The Cartesianspace position x_(s) is compared with a Cartesian space position x_(m)of the master and a positional deviation in Cartesian space is computedtogether with a force f_(m) required to move the master into a positionin Cartesian space which corresponds with the slave position x_(s) inCartesian space. The force f_(m) is then input to the master kinematicsconverter at 408 as indicated by arrow BA5.

From the f_(m) input, desired torque values are determined at 408. Thisis typically performed by a Jacobian Transpose (J^(T)) controller. Thetorque values are then input to the master output converter at 406 asindicated by arrow BA6. At 406, master electric motor currents i_(m) aredetermined from the torque values τ_(m) and are forwarded to the masterat 404 and as indicated by arrow BA7 to cause the motors to drive themaster to a position corresponding to the slave position.

Although the feedback has been described with respect to a new positiondesired by the master to track the slave, it will be appreciated thatthe surgeon is gripping the master so that the master does notnecessarily move. The surgeon however feels a force resulting fromfeedback Torques on the master which he counters because he is holdingonto the master.

The discussion above relating to the control system 400 provides a briefexplanation of one type of control system which can be employed. It willbe appreciated that instead of using a Jacobian Transpose controller, anInverse Jacobian Controller arrangement can be used. When using aninversed Jacobian controller, bilateral controller 410 may output aCartesian slave position command x_(sd) at AB5 to the kinematics module412, with the Cartesian slave position command indicating the desiredposition of the slave. Kinematics algorithm module 412 may then use, forexample, an inverse Jacobian algorithm to determine a desired jointspace position θ_(sd) which can be compared against the initial jointspace position of the slave θ_(s). From this comparison, joint torquesmay be generated to compensate for any positioning errors, with thejoint torques passed via AB6 to the slave input/output module 414 asdescribed above.

It should also be noted that control system 400 may couple actuation ofthe master handle (in the exemplary embodiment, variation of thegripping angle defined between grip members 218 as shown in FIG. 3B) toarticulation of the end effector (in the exemplary embodiment, openingand closing the end effector jaws by varying the end effector anglebetween end effector elements 102.1, 102.2 as illustrated in FIG. 7) inthe matter described above, by including the master grip input and theend effector jaw actuation in the joint and Cartesian positioneffectors, equivalent torque vectors, and the like, in the calculationswhich have been described.

It should be understood that additional controllers or controllermodules may be active, for example, to provide friction compensation,gravity compensation, active driving of redundant joint linkage systemsso as to avoid singularities, and the like. These additional controllersmay apply currents to the joint drive systems of the master and slaves.The additional functions of these added controllers may remain even whenthe master/slave control loop is interrupted, so that termination of themaster/slave relationship does not necessarily mean that no torques areapplied.

An exemplary controller block diagram and data flow to flexibly couplepairs of master controllers with manipulator arms are shown in FIGS.11A-11D. As described above, the operator 402 manipulates manipulators404, here inputting actuation forces against both the left and rightmaster manipulators f_(h) (L, R). Similarly, both left and rightpositions of the master input devices will also be accommodated by thecontrol system, as will forces and positions of four or more slavemanipulator arms f_(e) (1, 2, 3, and 4), x_(e) (1, 2, 3, and 4). Similarleft, right, and slave notations apply throughout FIGS. 11A-11D.

The encoder increments from each joint of the master input devices 404and the slave manipulators 416 are all input into a servocontrol inputpre-processor SCI. In some or all of the joints of the master or slavestructures, this information may be provided in an alternative format,such as with an analogue signal (optionally providing absolute positionindication) from a Hall effect transducer, a potentiometer, or the like.

Where at least some of the signals transmitted from master input devices404 or slave manipulators 416 comprise encoder increments, pre-processorSCI may include one or more accumulators 1002 as illustrated in FIG.11B. Positive and/or negative encoder increments are counted betweenservocycle transfer requests 1004, which are provided from a servotiming generator STG are accumulated in a first register 1006. Afterreceipt of transfer request 1004, the accumulated encoder incrementsfrom throughout the servocycle are transferred to second register 1008.

As schematically illustrated in FIG. 11C, the transfer request ispreferably offset from an encoder increment clock so as to avoidinadvertent encoder reading errors during servocycle data transfer. Inother words, to avoid losing encoder increments during data transfer, anasynchronous transfer request/encoder increment sample rate ispreferably provided, as illustrated in FIG. 11C. The sample rate willoften be higher than the rate at which the encoder can produceincrements, and the accumulators will generally hold incrementalposition information for all encoder-equipped freely moveable joints ofthe input and slave manipulators over a servocycle, the servocyclepreferably having a frequency of over 900 Hz, more preferably having afrequency of 1,000 Hz or more, often having a frequency of at leastabout 1,200 Hz, and ideally having a frequency of about 1,300 Hz ormore.

Preferably, an accumulator 1002 will be included in pre-processor SCIfor each encoder of the master input devices 404 and slave manipulators416. Each encoder accumulator will preferably accommodate at least a12-bit joint position signal, and in many cases will accommodate a14-bit joint position signal. Where analogue position signals areprovided, they will typically be converted to digital signals at orbefore storage in the pre-processor SCI, with as many as 48 jointsignals or more being provided in the exemplary pre-processor.

Referring now to FIGS. 11A and 11D, and first concentrating ontransmission to and from a first bilateral controller CE1 during aservocycle, joint positional information e_(m), e_(s) for a particularmaster input device 404/slave manipulator 416 pair is retrieved inresponse to a servointerrupt signal 1010 from the servo timing generatorSTG. The control processor CTP may transform these joint positionsignals to the desired coordinate reference frame, or may alternativelytransfer this information in joint space on to the bilateral controllerCE1 for conversion to the desired reference frame. Regardless, theposition is preferably transmitted from the control processor CTP to thebilateral controller CE1 using a direct memory access DMA controller orother high-speed data transmission system.

Once the positional information has been transferred from the controlprocessor CTP to controller CE1 at DMA interrupt 1012 (see FIG. 11D),the controller processes the positional information, comparing the endeffector positions in the surgical workspace with the input devicepositions (including both location and orientation) in the mastercontroller workspace.

As more fully explained in co-pending U.S. patent application Ser. No.09/373,678, filed Aug. 13, 1999, the full disclosure of which isincorporated herein by reference, the surgical and controller seworkspaces may be scaled and positioned relative to each other asdesired, often using positional information provided by the sensors ofthe set-up joints, and incorporating calibration and/or assemblyinformation of the master control console so as to identify the locationand/or orientation of the master input device relative to the viewer. Ingeneral, as the structure supporting the image capture device and endeffectors on the slave side are known, and as the location of the viewerrelative to the master input device can be calculated from similarknowledge regarding the lengths of the master input lengths, the mastercontroller joint angles, and the like, an appropriate coordinationtransformation may be derived so as to mathematically couple the masterspace of the master control workstation and the slave space in thesurgical environment. The information on both the master and slavelinkages in structure may be based on a model of these linkage andsupport structures, on design specifications for the linkage and supportstructures, and/or on measurements of individual linkages, which may bestored in a non-volatile memory of the slave and/or master control, suchas by burning calibration information into a memory of the appropriatestructure.

As illustrated in FIG. 11D, much of a servocycle time is used by thecontroller CE1 to calculate appropriate high-level instructions for themaster and slave systems. The results of these calculations aretransferred to control processor CTP via yet another DMA interrupt 1014.These high-level commands, typically in the form of desired forces to beapplied on the master and slave f_(m), f_(s) in a suitable referenceframe such as a Cartesian coordinate system are converted by the controlprocessor CTP to desired motor current signals, which are directed tothe appropriate motors by post-processor SCO.

While the pre- and post-processors, timing generator, control processor,and controllers are illustrated schematically in FIG. 11A as separateblocks, it should be understood that some or all of these functionalcomponents may be combined, for example, on a single processor board, orthat multiple processor boards may be used with the functions of one ormore of these components being separated on to separate processors.

As can be understood with reference to FIGS. 11A and 11D, while thefirst controller CE1 is processing the position and other informationassociated with the first master/slave pair, the pre- andpost-processors and control processor are processing and transferringdata for use by the second and third controllers CE2, CE3. Hence, theindividual controllers have asynchronous input and output times. Itshould be understood that more than three controllers may be providedfor additional master/slave pairs. In the exemplary embodimentillustrated in FIG. 11A, for example, the first and second controllersCE1 and CE2 might be dedicated to left and right hand inputs from thesurgeon, while the third controller CE3 may be used to move theendoscope using the left and/or right input device, or any other desiredinput system.

In the embodiment of FIG. 11A, servo timing generator STG includes amemory storing the master/slave pair assignments 1016. These pairassignments are communicated to the pre- and post-processors SCI, SCO,so that the information transferred to and from the control processorCTP is appropriate for the controller, and so that the commands from theappropriate controller are properly understood and transmitted to thedrive system for the appropriate joints. Reallocation of themaster/slave pair assignments is transmitted to the timing generator STGfrom the control processor CTP, and is then communicated from the timinggenerator to the pre- and post-processors during an intermittentinitialization phase, which may also be used to set up appropriateprocessor time intervals. Alternatively, the time intervals may befixed.

As should be understood by those of skill in the art, the flexiblemaster/slave pairing controller of FIG. 11A is still a simplification,and an appropriate controller will include a number of additionalsystems. For example, it is highly beneficial to include fault-checkingsoftware to ensure that all encoders or other joint sensors are readduring each servocycle, and that the drive systems of each driven jointof the master and slave are written to during each servocycle. If thefault-check is not successfully completed, the system may be shut down.Similarly, the control system may check for changes in pair assignments,for example, during data transfer to and/or from the camera controller.Similarly, pair assignments may be reviewed during and/or after a toolchange, during a left/right tool swap, when handing off tools betweentwo different master controllers, when the system operator requests atransfer, or the like.

It should be noted that the control system of FIGS. 11A-11D mayaccommodate flexible tool mountings on the various manipulators. Asdescribed above, the first and second controllers CE1, CE2 may be usedto manipulate tools for treating tissue, while the third controller CE3is dedicated to tool movements using inputs from both master inputdevices. In general surgical procedures, it may desirable to remove theendoscope or other image capture device from a particular manipulatorand instead mount it on a manipulator which was initially used tosupport a treatment tool. By appropriate commands sent via the controlprocessor CTP to the servo timing generator STG, the pair assignmentsfor the three controllers may be revised to reflect this change withoutotherwise altering the system operator's control over the system.

During pair re-assignment, appropriate data sets and/or transformationsreflecting the kinematics of the master/slave pairs, the relationship ofthe image capture device with the end effectors, and the like, may betransmitted to the controller. To facilitate swapping the image capturedevice from one manipulator to another, it may be beneficial to maintaina common manipulator structure throughout the system, so that eachmanipulator includes drive motors for articulating tools, endoscopeimage transfer connectors, and the like. Ideally, mounting of aparticular tool on a manipulator will automatically transmit signalsidentifying the tool to the control system, as described in co-pendingU.S. Patent Application Ser. No. 60/111,719, filed on Dec. 8, 1998,(Attorney Docket No. 17516-003210) entitled “Surgical Robotic Tools,Data Architecture, and Use.” This facilitates changing of tools during asurgical procedure.

A variety of adaptations of the exemplary control system will be obviousto those of skill in the art. For example, while the exemplaryembodiment includes a single master bus and a single slave bus, one orboth of these individual busses may be replaced with a plurality ofbusses, or they may be combined into a single bus. Similarly, while theexemplary servocycle time for an individual control pair is preferablyabout 1,000 msec or less, and ideally about 750 msec or less, the use ofhigher speed processing equipment may provide servocycle times which aresignificantly faster.

The master/slave interaction between master control station 200 and cart300 is generally maintained while the operator O is activelymanipulating tissues with surgical instruments associated with his orher left and right hands. During the course of a surgical procedure,this master/slave interaction will be interrupted and/or modified for avariety of reasons. The following sections describes selectedinterruptions of the master/slave control interaction, and are usefulfor understanding how similar interruptions and reconfigurations of thetelesurgical robotic network may be provided to enhance the capabilitiesof the overall robotic system. The exemplary interruptions include“clutching” (repositioning of a master control relative to a slave),repositioning of an endoscope, and a left-right tool swap (in which atool previously associated with a master control input device in a righthand of a surgeon is instead associated with an input device in a lefthand of the surgeon, and vice versa.) It should be understood that avariety of additional interruptions may occur, including during removaland replacement of a tool, during manual repositioning of a tool, andthe like.

Clutching.

In the course of performing a surgical procedure, the surgeon may wishto translationally reposition one or both of the master controlsrelative to the position or positions of a corresponding end effector oreffectors as displayed in the image. The surgeon's dexterity isgenerally enhanced by maintaining an ergonomic orientational alignmentbetween the input device and the image of the end effector. The surgeonmay reposition the master relative to the end effector by simplyinterrupting the control loop and re-establishing the control loop inthe desired position, but this can leave the end effector in an awkwardorientation, so that the surgeon repeatedly opens the control loop toreorient the end effectors for each translational repositioning.Advantageously, the ergonomic rotational alignment between input devicesand the images of the end effectors can be preserved after the mastercontrol or controls have been repositioned by a modified clutchingprocedure, which will now be described with reference to FIGS. 12 and13.

Referring to FIG. 12, a block diagram indicating the repositioning ofone of the master controls is indicated generally by reference numeral450 and will now be described. It will be appreciated that both mastercontrols can be re-positioned simultaneously. However, for ease ofdescription, the repositioning of a single master control will bedescribed. To reposition the master control relative to its associatedslave, the surgeon causes the control loop 400 linking master controlmovement with corresponding slave movement to be interrupted. This isaccomplished by activation by the surgeon of a suitable input device,labeled “Depress Master Clutch Button” at 452 in FIG. 12. It has beenfound that such a suitable input device can advantageously be in theform of a foot pedal as indicated at 208 in FIG. 2. It will beappreciated that any suitable input can be provided such as voicecontrol input, a finger button, or the like. It is advantageous toprovide an input device which does not require the surgeon to remove hisor her hands from the master controls so as to preserve continuity ofmaster control operation. Thus, the input device can be incorporated onthe master control device itself instead of having a foot pedal.

Once the input has been activated, e.g., by depressing the foot pedal,the control loop 400 between master and slave is interrupted. The slaveis then locked in its position, in other words in the position in whichit was at immediately before the foot pedal was depressed.

As can be described with reference to FIG. 11, upon depression of thefoot pedal, the link 410 in the control system 400 between master andslave is interrupted. The position in joint space of the slaveimmediately before depression of the foot pedal is recorded in a memoryof a slave joint controller indicated at 420 in dashed lines. Should aforce then be applied to the slave to cause it to displace to a newjoint position, the encoders on the slave relay signals to 414 where anew joint space position for the slave is computed and forwarded to theslave joint controller 420 as indicated by arrow BA9. This new jointspace position is compared with the joint space position in the memory,and joint space deviations are determined. From this joint spacedeviation, torques are computed to return the slave to the jointposition as recorded in the memory. These torques are relayed to 414 asindicated by arrow AB9 where corresponding electric motor currents aredetermined which are forwarded to the slave motors to cause it torestore its joint space position. Thus, the slave position is servolocked.

Referring again to FIG. 11, upon depression of the foot pedal at 452,the translational movement of the master is caused to float while itsorientation is locked, as indicated at 454 in FIG. 12. This step isachieved by a master Cartesian controller with memory as indicated at422 in FIG. 11. The functioning of the master Cartesian controller withmemory will now be described with reference to FIG. 13.

Upon activation of the foot pedal and repositioning of the master, thejoint space position input of the master control as indicated by θ_(a)is converted from joint space to Cartesian space at 406. From thisconversion, a Cartesian space position x_(a) of the master is obtained.The position in Cartesian space of the master immediately beforeactivation of the foot pedal is recorded in a memory at 424 and isindicated by x_(d). The current position x_(a) of the master as it movesto its new position is compared with the recorded position x_(d) at 456to obtain error signals, which correspond to positional deviations ofcurrent master position in Cartesian space when compared with therecorded position x_(d) in Cartesian space. These deviations or errorsare input to a feedback controller at 426 to determine a feedback forceto return the master to a position corresponding to the recordedposition x_(d). The components of the feedback force which correspondsto translational movement are zeroed at 428. Thus, translationalfeedback force components are zeroed and only orientational forcecomponents are forwarded from 428. The orientational force componentsare then converted to corresponding torques at 408, which are then inputto 406 (in FIG. 11) to determine currents for feeding to the electricmotors on the master to cause its orientation to be urged to remain in acondition corresponding to the orientation determined by x_(d). It willbe appreciated that the orientation at the position x_(d) corresponds tothe orientation of the slave since the slave continuously tracks themaster and the positions were recorded in memory at the same time Sincethe translational forces were zeroed, the translational movement of themaster is caused to float enabling the surgeon to translate the masterto a new, desired position. Such translational floating mayalternatively be provided by a variety of other methods. For example,the translational gains of controller 426 may be set to zero. In someembodiments, the translational elements of memory 424 may be continuallyreset to be equal to the input values x_(a), so that the differencebetween the measured position and the stored position is zero. It shouldalso be understood that despite the zeroing of the translational terms,additional controller functions such as friction compensation, gravitycompensation, or the like, may remain unaltered.

Referring again to FIG. 12 of the drawings, when the master controlshave been moved to their desired position the foot pedal is released.Upon release, the translational deviations relating to the new positionof the master control relative to its associated slave is incorporatedinto 410 to define a new Cartesian space position at which the slaveposition corresponds to the master position. In particular, thetranslational derivations may be incorporated in the fixed offsetsdescribed above, preferably using the algorithm described herein toavoid inadvertent sudden movements or forces.

Since the orientation of the end effector was held at the same position,and since the master orientation was caused to remain in a correspondingorientation, realignment of the end effector and master is normally notnecessary. Re-connection of master and slave takes place upon release ofthe foot pedal as indicated at 456. The re-connection will now bedescribed with reference to FIG. 19.

Referring to FIG. 19, a block diagram illustrating the steps involved inre-connecting the control system 400 between the master and the slave isgenerally indicated by reference numeral 470.

The first step involving in re-connecting control between the master andthe slave, and as indicated at 472, is to determine whether or not themaster orientation is sufficiently close to the slave orientation. Itwill be appreciated that it could happen that during repositioning ofthe master as described above, the surgeon could be urging the pincherformation on the master away from its orientationally aligned positionrelative to the slave. Should re-connection of control between masterand slave then occur, it could result in reactive motion by the slaveresulting from the urging force applied by the surgeon on the pincherformation. This reactive motion by the slave could cause unnecessarydamage to organs, or tissue, or the like, at the surgical site andshould be avoided. Accordingly, at 472 the orientation of master andslave is compared. If the orientation of the master does not coincidewith the orientation of the slave or does not fall within an acceptableorientational deviation, re-connection of control between master andslave will not be enabled. In such a case an appropriate message istypically displayed on the viewer indicating to the surgeon that arequired corrective action is required to cause the orientation of themaster to be within the acceptable deviational range relative to theorientation of the slave. An example of such a message is one indicatingto the surgeon to relax his or her grip on the pincher formation.Simultaneously, the master alignment algorithm may be executed asdescribed hereinbelow with reference to FIG. 18.

When the orientations of master and slave are sufficiently similar, theslave orientation is optionally snapped with the master orientation inCartesian space as indicated at 474. Once the orientation is snapped,the Jacobian Inverse controller on the slave is enabled as indicated at476. Thereafter, the Cartesian force reflection commands and gains aredownloaded as indicated at 478.

As used herein, the snapping of the slave orientation to the masterorientation means that the orientational offsets in bilateral controller410 are reset to zero, so that the master and slave orientations begintracking each other. In synchronization with this snapping, the controlsystem 410 is reconfigured to normal bilateral control, preferably usinga Jacobian inverse, as indicated at step 476. The appropriate commandsand gains are downloaded as indicated at step 478.

In many embodiments, rather than instantaneously snapping the master tothe slave, the orientational offsets in bilateral controller 410 mayalternatively be slowly and smoothly reduced to zero, thereby providinga smoother transition between operating modes. This may be effected by,for example, filtering the orientational offset values to zero.

In general, some and/or all transition of control system 400 betweenoperating configurations or modes, including those described withreference to steps 454 and 456 of the master repositioning algorithm ofFIG. 12, as well as a variety of similar steps described hereinbelow,may include potentially substantially instantaneous changes inconfiguration or perimetric values of the control system. For example,interrupting or opening the loop of bilateral controller 410, enablingmaster Cartesian controller 422, resetting memory 424 or the controllergains in P.I.D. controller 426 might be performed by substantiallyinstantaneously changing the perimetric values and/or configurations.Such instantaneous changes may be fundamentally different than normalmaster/slave operation, where the computations are continually repeatedusing fixed perimetric values and operational configurations, with onlythe sensor readings changing.

Where substantially instantaneous changes in perimetric values and/orconfiguration are imposed, it is possible that a sudden change in motorcurrents may result, causing the system to jerk. Such inadvertentinstantaneous movements of the system may be transmitted to the surgeonor other system operator, and can be disconcerting and/or reduce theoverall feel of control the operator has over the system. Additionally,unexpected rapid movements of a surgical instrument at a surgical siteare preferably minimized and/or avoided. Hence, rather than effectingthese changes in perimetric values and/or configuration instantaneously,the changes will preferably be timed and executed in a manner so as toavoid significant instantaneous changes in the computed motor currentsapplied before, during, and after the change in configuration. Thissmooth change of perimetric values and/or controller configurations maybe provided by a “no-jerk” algorithm which will be described withreference to FIG. 19A.

The relevant control system mode transitions typically involve aconfiguration change, a change in a fixed memory value, or the like. Inparticular, bilateral controller 410 makes use of fixed offsets in itsmemory. Controllers 420, 422, and 560 also contain fixed commands intheir memories. The no-jerk algorithm, which generally decreases and/oreliminates rapid inadvertent movement of the master or slave, utilizesknown sensor readings, configuration information, and memory valuesimmediately before a control system operating mode transition. Byassuming that sensor readings will remain predictable, changing onlyslightly during the controller mode transition, the no-jerk algorithmcomputes desired memory reset values by also taking into account theknown end values or configuration, and by synchronizing the change invalues so as to promote smooth motor current changes during the modetransition. For some uses, the no-jerk algorithm my reduce or eliminatesudden changes in motor torques by using pre-transition (and optionallyfiltered) motor currents or joint torque values in place of or incombination with the pre-transition sensor configuration and memoryvalues as inputs.

Referring now to FIG. 19A, pre-transition configuration, perimetricvalues, and memory values are used, together with sampled pre-transitionsensor values 702 to compute pre-transition joint torques at step 704.Alternatively, these pre-transition joint torques may be directlyobserved, optionally with filtering, at step 706. Regardless,post-transition joint torque values are forced to match thepre-transition joint torque values at step 708. Meanwhile, using knownpost-transition configuration and perimetric values 710, thepost-transition effective feedback gains may be determined at step 712.These post-transition effective feedback gains may be inverted and usedtogether with the post-transition joint torques to calculate a desiredpost-transition error signal at step 714. The post-transition sensorvalues may be predicted at step 716. These post-transition sensor valuesmay be estimated by assuming that smooth sensor readings will beprovided, and knowing the time it takes to effect transition.

The desired post-transition error signal and predicted sensor values maybe used to derive a desired post-transition command signal at step 718.

Based on the known post-transition configuration, the post-transitioncommand signal will generally determine the desired memory or offsetvalue through calculations performed at step 720. This post-transitionmemory or offset value is reset in synchronization with the transitionat step 722. Hence, once the desired mode transition is input,information about the configuration of the system before and after thechange takes place allows smoothing of the transition.

Repositioning of one of the slaves relative to one of the masters willnow be described with reference to FIGS. 14 and 15. It is to beappreciated that both slaves can be repositioned relative to theirassociated masters simultaneously. However, for ease of explanation therepositioning of a single slave relative to its associated master willnow be described.

In FIG. 14 a block diagram indicating steps involved in repositioning aslave relative to its associated master is generally indicated byreference numeral 500. When it is desired to move the end effector of aslave to a new position, a suitable input is activated to interrupt thecontrol loop 400 between the master and the slave. Such a suitable inputcan be in the form of a button on the robotic arm as indicated at 480 inFIG. 5A. Depressing such a button to interrupt the control loop 400 isindicated by the term “Depress Slave Clutch Button” at 504 in FIG. 14.Once the button is depressed, the control between master and slave isinterrupted to cause the translational movements of the slave to floatwhile the orientation of the end effector is locked as indicated at 502in FIG. 14.

In general, when movements of one or more joints of a master or slavelinkage are allowed to float, the floating joints may optionally stillhave some forces imposed against the joint by their associatedjoint-drive systems. More specifically, as described more fully inco-pending U.S. patent application Ser. No. 09/287,513, the fulldisclosure of which is incorporated herein by reference, the controllermay impose actuation forces on the master and/or slave so as tocompensate for gravity, friction, or the like. These compensation forcesmay be maintained on the floating joint or joints even when the controllink for actuating the joint is otherwise open.

The step indicated at 502 will now be described in greater detail withreference in particular to FIG. 15, and also with reference to FIG. 11.When the button 480 is depressed, the position θ_(d) of the slave injoint space immediately before depression of the button is recorded in amemory of the slave joint controller 420, and as indicated at 460. Asthe slave is moved thereafter, its position in joint space indicated byθ_(a) is compared with θ_(d) at 462. As θ_(a) deviates from θ_(d) errorsignals corresponding to the positional deviation in joint space isdetermined at 462 and is passed to 464. At 464 required torques for theelectric motors on the slave are determined to cause the slave to returnto the θ_(d) position. The torques thus determined which relate totranslational torques of the slaves are zeroed at 466 to permit theslave translational movements to float. The torques corresponding toorientational movement are not zeroed. Thus, any environmental forces onthe end effector urging an orientational position change are fed back tothe end effector to cause it to retain its orientation. In this way theorientation of the end effector relative to the end of the instrumentshaft 104 is locked in position. Although the orientation of the endeffector does not change relative to the end of the shaft, it doeschange in position in Cartesian space as a result of translationalposition change. It should be understood that zeroing of the outer jointtorques at step 466 may be effected by a variety of methods, includingzeroing of the appropriate gains in P.I.D. controller 464, continuallyupdating the appropriate elements in memory 460 so as to compute a zeroerror signal at comparison 462, or the like.

It should be also be understood that a variety of additional operationconfigurations may be implemented which allow slave transitionalmovements to float free of the master control. For example, slavetransitional forces may be zeroed in Cartesian space (analogous to themaster clutching algorithm described with reference to FIGS. 12 and 13).Alternatively, control system 400 and/or bilateral controller 410 may beinterrupted only for translational motions, locking the mastertranslational position and allowing the slave to float in transnationalposition, all while connecting the master orientation to the slaveorientation. Once the slave is at the desired position the button isreleased as indicated at 510 in FIG. 14.

When the button is released, the master orientation is re-aligned withthe slave orientation as indicated at 512. The re-aligning of theorientation of the master and slave is now described with reference toFIG. 18. The steps involved in such re-alignment are generally indicatedby reference numeral 550.

At 552 the slave position θ_(s) in joint space is read. The positionθ_(s) is then converted to a position x_(s) in Cartesian space at 554using slave forward kinematics. Thereafter at 556, the desiredorientation of the master is set to equal the slave orientation inCartesian space. Thus x_(m), the master orientational position inCartesian space is set to equal x_(s), the slave orientational positionin Cartesian space. Thereafter at 558, inverse master kinematics isemployed to determine the master joint position θ_(m) in joint spacewhich corresponds to x_(m), the master position in Cartesian space.Finally, the master is then caused to move to θ_(m) by causingappropriate signals to be sent to the motors on the master as indicatedat 560. It will be appreciated that the surgeon will generally releasethe master to enable it to move into an orientation aligned with theslave orientation.

Referring again to FIG. 14, after the re-alignment step at 512, themaster is reconnected to the slave as indicated at 513. It will beappreciated that the step 513 is the same as that described above withreference to FIG. 19. The master realignment is described in more detailin Application Ser. No. 60/116,842.

Endoscope Movement.

Referring now to FIGS. 11, 16, and 17, repositioning of the endoscope tocapture a different view of the surgical site will now be described. Asthe surgeon may wish to view the surgical site from another position,endoscope arm 302 can selectively be caused to vary its position so asto enable the surgical site to be viewed from different positions andangular orientations. The arm 302 includes appropriately positionedelectrical motors controllable from the control station 200. Theendoscope arm can thus be regarded as a slave and is typicallycontrollable in a control loop similar to that shown in FIG. 11.Regarding the endoscope as another slave, cart 300 has three slaves, therobotic arm assemblies 310 and 304, and two masters 210.

To vary the position of the endoscope, the surgeon activates an input atthe control station 200. The input can be generated from any appropriateinput device, which can include a depressible button, or a voice controlsystem, or the like. Upon such activation, the control loops betweenmaster 210 and slaves 310 one of which is indicated in FIG. 11, areinterrupted and the parts of the control loop on both master sides areoperatively linked to a dormant control loop portion similar to that ofthe slave in FIG. 11, but which is arranged to control endoscope armmovement. The surgeon can then change the position of the endoscope toobtain a different view of the surgical site by means of manual inputson the master controls 210. When the endoscope has been moved to adesired position, control between master and slave is re-established inaccordance with the methods described above including automaticassessment of left and right hand allocation between masters and slavesas already discussed.

An exemplary method and system for robotic movement of the endoscopeusing both of the master controllers is described in more detail inApplication Ser. No. 60/111,711, filed on Dec. 8, 1998, and entitled“Image Shifting for a Telerobotic System,” the full disclosure of whichis incorporated herein by reference.

At times, such as when the scope is moved to an alternative minimallyinvasive aperture, or when a scope is removed and replaced, theendoscope may be manually positioned. The steps involved inrepositioning the endoscope are indicated by reference numeral 600 inFIG. 16. To do this a suitable input device is activated.

The suitable input device is typically in the form of a depressiblebutton on the endoscope arm 302. However other methods such as voicecontrol or the like can be used instead. The button is similar to thebutton on the arm 312 as described above. The depressing of such abutton is indicated at 602 in FIG. 16 and is labeled “Depress cameraslave clutch button”. Upon activation of the input button the toolslaves and masters are servo locked at the positions they were atimmediately before activation of the input button.

When the button is depressed, all the joints on the endoscope arm 302are caused to float as indicated at 609. This will now be described ingreater detail with reference to FIG. 17. As soon as the button isdepressed, the position of the endoscope in joint space immediatelybefore depression of the button is recorded as indicated by θ_(d). Whenthe endoscope arm 302 is then moved to a new desired position, itspresent position indicated by θ_(a) in FIG. 17 is compared with θ_(d) at604 to determine joint positional errors or deviations. These errors arepassed to 606. The torques then determined are zeroed at 608 to causethe joints on the endoscope arm to float to enable repositioning.

It will be appreciated that floating the endoscope arm can also beachieved by setting the gains in 606 to zero or continually updatingθ_(d) to watch θ_(a) as to compute a zero error signal at 604. Similarlyone might disable the endoscope arm controller altogether or zero themotor commands.

It will also be appreciated that the endoscope could be freed to move intranslation while locked in orientation, analogous to theabove-described methods. Furthermore, one could control the orientationto keep the image aligned with horizontal or vertical, that is keep thetop of the image facing upward (for example, so that gravity isconsistently downward in the image shown to the system operator), whilefloating translational degrees of freedom. Again this is analogous tomethods described above, and can be used to disable and/or float aspectsof the endoscope controller in Cartesian space.

When the endoscope arm is brought into the required position the buttonis released as indicated at 610 in FIG. 17. Thereafter the masters arerealigned with the slaves as indicated at 612 and as already describedwith reference to FIG. 18. Thereafter at 614 control between master andslave is re-established and as already described with reference to FIG.19.

Though the above algorithms for repositioning masters, slaves and/or theendoscope arm were described in isolation, they can also be executed inparallel, allowing for simultaneous repositioning of any number ofsystem components.

Left-Right Tool Swap.

Referring now to FIG. 20 of the drawings, in which like referencenumerals are used to designate similar parts unless otherwise stated, animage as viewed by the surgeon, and as captured by the endoscope, isgenerally indicated by reference numeral 800.

During the course of a surgical procedure, the surgeon is oftencontrolling the actions and movements of the end effectors by inputtingmanual movements and actions on the master controls while viewing thecorresponding end effector movements and actions in the image displayedon the viewer. The left hand master control is typically operativelyassociated with the end effector displayed on the left hand side of theimage and the right hand master control is operatively associated withthe end effector displayed on the right hand side of the image.

As described above, the surgeon may wish to perform an image shift bymoving the viewing end of the endoscope relative to the surgical site toview the surgical site from a different position or angle. It couldhappen that during the conducting of the surgical procedure, such assubsequent to an image shift, the end effector which was on the left ofthe displayed image is now on the right, and similarly the end effectorwhich was on the right of the displayed image is now on the left.Furthermore, during the course of, e.g., training, or the like, anoperator of the minimally invasive system may wish to operativelyassociate the two master controls with a single end effector so as toenhance a training procedure of the system. This invention provides aminimally invasive telesurgical system which provides for selectivelypermitting operative association of any one or more of a plurality ofmaster controls with any one or more of a plurality of end effectors.

The image 800 is schematically indicated in FIG. 21 at an enlargedscale. The image 800 indicates the end effectors 102 at the working ends110 of two surgical instruments similar to the surgical instrument 100shown in FIG. 6. In the image, the portions of the shafts 104 of thesurgical instruments extend outwardly from the image on respectively aright hand side and a left hand side of the image. Referring again toFIG. 20 of the drawings, the master control device 210 on the right handside of the surgeon is operatively associated with the slave includingthe medical instrument defining the shaft extending outwardly toward theright hand side of the image 800. Similarly, the master control device210 on the left hand side of the surgeon is operatively associated withthe slave including the medical instrument defining the shaft extendingoutwardly toward the left hand side of the image 800. Accordingly, ananthropomorphic or immersive surgical environment is created at theworkstation 200 and the surgeon experiences an atmosphere of directlycontrolling actions and movements of the end effectors 102.

In FIG. 21A, the end effectors are shown to be at different positions inthe image. However, it is still clear which shaft 104 extends outwardlyto the left and right of the image. Accordingly, the same associationbetween the master control devices and the slaves prevails.

Referring now to FIG. 22 of the drawings, an image shift has takenplace. This can happen, for instance, where the surgeon wishes to changethe orientation of the surgical site as viewed through the viewer. Thiscan be accomplished by causing the endoscope to displace angularly aboutits viewing axis. It will be appreciated that the endoscope mounted onthe robotic arm 302, as can best be seen in FIG. 5, can be caused todisplace angularly about its viewing axis from the control station 200.

The new image 802 shown in FIG. 22 for the sake of example, was broughtabout by such angular displacement of the endoscope. Accordingly, theimage 802 has undergone an angular displacement, as indicated by arrow804. The shaft of the medical instrument which extended to the right ofthe image now extends to the left of the image and the shaft of themedical instrument which extended to the left of the image now extendsto the right of the image. If the association between masters and slaveswhich existed immediately before the image shift was to prevail, thiswould severely impede the surgeon's ability to carry on with thesurgical procedure since left hand control would be associated withright hand actions and movements of the end effector as displayed on theviewer, and vice versa.

To compensate for such a situation, the minimally invasive surgicalsystem of the invention causes the association between masters andslaves which prevailed immediately before the image shift, to beinterrupted and then to be switched or swapped automatically. Once thishas taken place, master control on the surgeon's right hand side isassociated with the slave which includes the shaft extending outwardlyto the right of the new image and the master control on his or her lefthand side is associated with the slave defining the medical instrumenthaving the shaft which extends outwardly to the left of the new image.Thus, the anthropomorphic surgical environment is retained at thecontrol station 200.

Referring now to FIG. 22B of the drawings, the steps involved in causingthe association between a master and a slave to be swapped with theassociation of another master and slave will now be discussed.

The first step, indicated by reference numeral 900 in FIG. 22B of thedrawings, is to determine the positions of the remote centers orfulcrums 349 of the slaves relative to a Cartesian space coordinatesystem having its origin at the viewing end of the endoscope. This stepwill now be described in greater detail and with reference to FIG. 10 ofthe drawings.

In other words, in the above discussion, and throughout the remainder ofthe following discussion, the remote centers or fulcrums 349 areconsidered coincident with the port of entry (as is typical in thepreferred embodiment). In other embodiments, however, these points maynot coincide (or even exist, for example, when a distal portion or anendoscopic tool is free to pivot above the insertion point, relying onthe tendency of the tool to pivot at this point with no remote centerimposed) in which case all calculations should be based on the locationof the port of entry. It will also be appreciated that the system maydetermine these port locations from sensor information and pre-existingknowledge of the cart 300, set-up joints, and manipulator arms.Alternatively, the locations could be determined by other sensors or byprocessing the image 800 directly to observe and extrapolate the pivotpoints of the displayed tool shafts.

As described above with reference to FIG. 10, the positions of eachfulcrum are generally determined relative to the Cartesian coordinatesystem 902, optionally using sensors of the set-up joints. This isindicated at step 911 in the method of FIG. 22B after which (at step913) a determination is made as to whether or not the (X,Y) positions ofeach fulcrum are sufficiently spaced apart relative to each other topermit the minimally invasive surgical system of the invention todetermine a left hand and right hand allocation for the robotic armassemblies or slaves. This step will now be described in greater detail.

It will be appreciated that the cart or trolley 300 and the robotic aimassemblies 395, 310, and 302 mounted thereon are not mechanicallyperfect structures. Thus, in computing the (X,Y) coordinates for eachfulcrum 349 positional errors can arise due to, e.g., external forcessuch as gravity, mechanical misalignments, miscalibration and the like.The range of such positional errors which can arise is indicated in FIG.22A. FIG. 22A indicates the x-x and y-y axes of the coordinate system902. The circular part of the shaded area in FIG. 22A represents an areacorresponding to an error range or margin resulting from such errors asdescribed above. The parts of the shaded area diverging outwardly alongthe x-x axis and from the circular part represent regions where thepositions of the fulcrums are too close to the x-x axis for anappropriate allocation to be made.

To determine whether or not the (X,Y) positions of the fulcrums 349 fallin the shaded area, a midpoint between the (X,Y) positions istransformed onto the x-x and y-y axis as indicated in FIG. 22A such thatthe midpoint coincides with the origin 904. With reference again to FIG.22B of the drawings, should the positions of the fulcrums 349 falloutside the shaded error region, the next step as indicated by referencenumeral 915 is performed. If not, an alternative method to allocate leftand right position is followed as indicated by the step 917, as furtherdescribed herein below.

The step 915 involves a selection or allocation of a right hand and lefthand position to the slaves. Accordingly, the slave defining the fulcrumto the left of the x-x axis in FIG. 22A is assigned the left handposition and similarly the slave defining the fulcrum to the right ofthe x-x axis is assigned the right hand position.

When this allocation has been made the step indicated at 919 isperformed. The step at 919 involves making a comparison between theallocated left and right hand positions with a previous left and righthand allocation. Should these allocations be the same, the associationbetween masters and slaves stays as it was as indicated at 921. Shouldthe allocation not be the same, the step indicated at 923 is performed.

The step 923 involves requesting a swap between the master and slaveassociations and will now be described with reference to the blockdiagram shown in FIG. 22C.

When performing a swap the control loops between the masters and slavesare temporarily interrupted as indicated at 925. This will now bedescribed with reference to FIG. 11 of the drawings. It will beappreciated that the control loop 400 indicates a single control loopwhich operatively associates a single master with a single slave. Theslave side of the loop is indicated below the dashed line in FIG. 11 andthe master side of the loop is indicated above the dashed line. It willbe appreciated that a similar control loop is provided for the othermaster and slave pair. The control loop of each master and slave pairare interrupted at the bilateral controller in step 925. Upon suchinterruption the positions of the masters and slaves are locked inposition by means of the respective master and slave joint controllers420, 560 in the case of each master and slave pair.

Referring again to FIG. 22C, after interruption of the control loops,the surgeon is then informed that a swap is about to take place at step927. This step typically involves causing a message to be displayed inthe image at the viewer. The message can require that the surgeonprovide an input to acknowledge his or her awareness of the swap to takeplace. Such an input can be generated in any appropriate manner such asupon depression of a button, or by means of voice control, or the like.When such an input is generated, operative association between eachmaster and its new associated slave is then established at step 929.Thus, referring once again to FIG. 11 of the drawings, the master sideof the control system 400 is linked to the slave side of the othercontrol loop and likewise the master side of the other control loop islinked to the slave side of the control loop 400.

Once the control loops have been connected, each master is moved intoalignment with its new associated slave at step 931, as described withreference to FIG. 18. Each master can then be connected with its newassociated slave at step 933, as described with reference to FIG. 19 ofthe drawings. Once these steps have been performed, operative controlbetween each master and its new slave is fully established as indicatedat 914 in FIG. 22B.

Returning now to FIG. 22B of the drawings, and where the positions ofthe fulcrums 349 fall within the error margin as indicated in FIG. 22Aas determined at 911 in FIG. 22B, the step indicated at 917 will now bedescribed. At 917, an alternative method of determining positions of thefulcrums is employed. This step involves determining the orientation ofthe endoscope relative to the cart 300. To determine the orientation ofthe endoscope relative to the cart the positional sensors are employedto determine whether the viewing end of the endoscope is directed towardor away from the cart. Should the end of the endoscope be directed awayfrom the cart, the right hand slave is automatically allocated a righthand position and the left hand slave is automatically allocated a lefthand position at the step 935, this allocation presuming a direction ofview as indicated by arrow K in FIG. 4. Should the viewing end of theendoscope be directed toward the cart, the left hand slave is allocateda right hand position and the right hand slave is allocated a left handposition at the step 935. Again, allocation presumes a direction of viewas indicated by the arrow K in FIG. 4. This method is based on thepresumption that set up joints, indicated by reference numerals 395 inFIG. 9 do not readily cross each other.

It will be appreciated that the endoscope arm 302 can selectively becaused to vary its position so as to enable the surgical site to beviewed from different positions and angular orientations. The arm 302includes appropriately positioned electrical motors controllable fromthe control station 200. The endoscope arm can thus be regarded as aslave and is typically controllable in a control loop similar to thatshown in FIG. 11.

Both masters can optionally be operatively associated with a singleslave, e.g., for training purposes. Employing the methods describedabove will also enable a surgeon selectively to control any one or moreof these multiple slave arms with only two masters. Furthermore, twocontrol stations can be operatively associated with a single cart 200.Thus one master of each control station can then be operatively linkedto a single slave arm and each other master control with a single otherslave arm. This can be advantageous for e.g., training purposes, or thelike.

Regarding the endoscope as another slave, the minimally invasivesurgical system of the invention accordingly has three slaves, therobotic arm assemblies 310 and 304, and two masters 210. As describedherein, further slave arms may be incorporated as an optional feature.

It will be appreciated that the allocation steps described above forallocating master and slave association are typically automaticallycarried out at the commencement of a surgical procedure after the slaveshave been brought to initial starting positions at the surgical site.Naturally, in addition, or instead, the allocation steps can beinitiated manually when appropriate by activating a suitable input toinitialize the allocation steps. The steps are also automaticallycarried out when either one or both masters are repositioned relative tothe slaves, when either one or both slaves are repositioned relative tothe associated master or masters and when the endoscope is repositioned,as described earlier in this specification. It is to be appreciated thatwhere an input is required in this specification and where appropriatesuch an input can be by way of any suitable input, such as buttons,cursor selection foot pedal toggling, voice control or any othersuitable form of input.

It will furthermore be appreciated that the determination ofmaster-slave association, which is computed automatically according toFIGS. 22A and 22B, may be specified manually by way of a suitable inputdevice, such as buttons, a foot pedal, voice control, mouse input, orany other suitable form. If the association is specified manually, onlysteps 911 and 916 need be performed to execute the association.

In a system with more than two masters or more than two robotic armswith associated instruments, master-slave association will preferably beentered manually. This can be accomplished by interrupting the currentassociation to allow the master to translate freely as described abovewith reference to FIGS. 12 and 13, then using the floating master as amouse-like pointing device to highlight and/or select the image of oneof the slaves. To complete the process, the master is locked and the newassociation activated using steps 919 and 923. Any slaves 9 that are notpart of an existing association are locked in joint space usingcontroller 420. The slave location at the time of disassociation isstored in memory in 420, and compared against the sensor signals toprovide appropriate feedback torques.

Similarly, in a system with more masters than slaves, only mastersselected by appropriate input devices are associated with slaves, whilethe remainder are locked using a controller such as controller 560.

Robotic Network.

Referring now to FIGS. 1, 23A and 23B, many of the above steps may beused to selectively associate any of a plurality of tools with any of aplurality of input devices. Operator O may initiate a tool selectionsubroutine 910 by actuating a tool selector input, such as by depressingfoot activated button 208 a of workstation 200 (illustrated in FIG. 2).Assuming operator O is initially manipulating tools A and B with inputdevices 210L and 210R using his or her left and right hands LH and RH,respectively, tool selector procedure 910 will be described withreference to a change of association so that input device 210L isinstead associated with a tool C, here comprising a tissue stabilizer120.

Once the tool selector subroutine is activated, the operator willgenerally select the desired tools to be actively driven by the roboticsystem. The surgeon here intends to maintain control over Tool B, butwishes to reposition stabilizer 120. Optionally, operator O will selectbetween the left and right input devices for association with the newlyselected tool. Alternatively, the processor may determine theappropriate left/right association based on factors more fully describedin co-pending U.S. Patent Application Ser. No. 60/116,891, filed on Jan.22, 1999, and entitled “Dynamic Association Of Master And Slave In AMinimally Invasive Telesurgical System,” (Attorney Docket No.17516-004700) the full disclosure of which is incorporated herein byreference.

Optionally, operator O may select the desired tools for use bysequentially depressing selector input 208 a, with the processorsequentially indicating selection of, for example, Tools A and B, then Band C, then A and C, and the like. Controller station 200 may indicatewhich tools are selected on display 800, audibly, or the like. Forexample, the image of the selected tools viewable by the surgeon may becolored green to designate active manipulation status, and/or thedeselected tools may be colored red. Preferably, any deselected tools(for example, Tool A) will be maintained in a fixed position per step914. The tools may be held in position using a brake system and/or byproviding appropriate signals to the drive motors of the tool and armactuation system to inhibit movement of the tool. The tool fixation step914 will preferably be initiated before a master input device isdecoupled from the tool, so that no tool moves absent an instructionfrom an associated master. Tool fixation may occur simultaneously withtool selection. The selected master may be allowed to float, step 916,during and/or after tool fixation and tool selection.

Once the selected master has been allowed to float, the master may bemoved into alignment with the selected tool as illustrated in FIG. 23A,as was described above with reference to FIG. 18. Often, this will occurwhile the surgeon keeps a hand on the input device, so that the drivemotors of the master should move the master at a moderate pace and witha moderate force to avoid injury to the surgeon. Master input device210L may then be coupled to tool C (stabilizer 120 in our example) whiletool A is held in a fixed position. This allows the operator toreposition stabilizer 120 against an alternative portion of coronaryartery CA. The tool selection process may then be repeated tore-associate the masters with tools A and B while tool C remains fixed.This allows the surgeon to control repositioning of stabilizer 120without significantly interrupting anastomosis of the coronary artery CAwith the internal mammary artery IMA.

A number of alternative specific procedures may be used to implement themethod outlined in FIG. 23B. Optionally, the interface may allow theoperator to manually move the input devices into apparent alignment withthe desired tools while the tool selector button is depressed. In FIG.23A, the surgeon might manually move master 210L from alignment withtool B into approximate alignment with tool C. The processor could thendetermine the tools to be driven based on the position of the inputdevices when the button is released, thereby allowing the operator to“grab” the tools of interest. Some or all of the tools (Tools A, B, andC) may optionally be maintained in a fixed configuration when theoperator is moving the master controllers to grab the tools.

Allowing an operator to sequentially control more than two robotic toolsusing the operator's two hands can provide significant advantages. Forexample, referring again to FIG. 1, by allowing operator O the abilityto select in real time and control any one or two tools 100 of cart 300and auxiliary cart 300A, the surgeon will often be able to act as his orher own assistant.

In addition to allowing the operator to safely reposition a stabilizer100 against a coronary artery and the underlying beating heart duringbeating heart coronary artery bypass grafting, a variety of alternativeprocedures would also be facilitated by such capabilities. As anotherexample, in the procedure of gall bladder removal (cholecystectomy), thesurgeon will generally want to first to provide exposure of the organ(retraction) to expose the area of interest. This generally involvesguiding a retractor tool (mounted, for example, to a first manipulatorarm) to expose an area of interest. The area of interest may be exposedfor viewing through an endoscope mounted, for example, to a secondmanipulator arm. The surgeon might thereafter want to use two hands todirect tools in dissecting tissue covering the cystic duct and arterywhile the retractor remains stationary. One of the two tools (which maybe mounted on third and fourth manipulator arms) can be used to stretchthe tissue (traction or grasping) while the other tool is used to cuttissue (sharp dissection) to uncover the vessel and duct structures.Hence, the ability to selectively control four manipulators from asingle console allows the surgeon to control the manipulation,retraction/stabilization, and viewing angle of the procedure, withouthaving to verbally instruct an assistant.

At any time during the dissection, the surgeon could have the capabilityof adjusting the exposed area of the cystic duct by again selectivelyassociating a master input device in his or her left or right hand withthe retractor. Once the desired change in exposure is obtained byrepositioning the retractor, the surgeon can deselect the retractiontool, and then select and move the endoscope to a more appropriateviewing angle for work on the newly exposed tissue. Thereafter, thesurgeon can again select the grasping and cutting tools to manipulatethe tissues using both hands.

The ability to control four or more surgical arms also gives the surgeonthe capability of selecting from among alternative tools based on toolfunction and/or anatomical constraints. For example, tools A, B, and Cmay all have end effectors comprising universal graspers. If the surgeonis afforded a better approach to tissue dissection by using themanipulator arms associated with tools A and B in certain parts of atwo-handed dissection procedure, but would prefer to use tools B and Cfor alternative portions of the two-handed dissection procedure, theoperator is free to switch back and forth between tools A and C usingtool selection subroutine 910. Similarly, if a cauterizing electrodeblade is desired intermittently during a dissection, the operator mayswitch back and forth between tools A and C to dissect, and thencauterize, and then dissect, etc., without having to wait for anassistant to repeatedly swap tools.

Advantageously, providing a “redundant” manipulator may reduce the needfor a laparoscopic surgical assistant who might otherwise be called onto perform intermittent functions by manually manipulating a tool handleextending from an aperture adjacent the manipulator aims. This can helpavoid interference between manual tools, personnel, and the movingmanipulator arms, and may have economic advantages by limiting thenumber of highly skilled personnel involved in a robotic surgicalprocedure. The procedure time may also be decreased by avoiding the timegenerally taken for a lead surgeon to verbally direct an assistant.

Tool Hand-Off.

Many of the steps described above will also be used when “handing-off”control of a tool between two masters in a tool hand-off subroutine 920,as illustrated in FIG. 24. Tool hand-off is again initiated by actuatingan appropriate input device, such as by depressing foot pedal 208 bshown in FIG. 2.

The tool to be transferred will typically be designated, again using anyof a variety of designation input methods or devices. The transfer toolmay be coupled to any master input device or devices, including an inputdevice of master control station 200, assistant control station 200A, orauxiliary input 12 of auxiliary cart 300A (as illustrated in FIG. 1).Optionally, the input device which will assume control of the designatedtool is also selected in designation step 922, although selectionbetween left and right masters may again be left to the processor, ifdesired.

Once the tool and master are designated, the hand-off tool (and any toolpreviously associated with the designated master) is fixed, and thedesignated master is allowed to float. The master is then aligned andconnected with the tool as described above.

Camera Switch and Right Access Robotic CABG.

The following pertains to an exemplary robotic surgery procedure thatmay be performed with the foregoing apparatuses and methods. Referringnow to FIGS. 1, 25A, and 25B, a single complex minimally invasivesurgery will often involve interactions with tissues that are bestviewed and directed from different viewing angles. For example, inperforming a Coronary Artery Bypass Grafting (CABG) procedure on patientP, a portion of the internal mammary artery IMA will be harvested fromalong the internal surface of the abdominal wall. The internal mammaryartery IMA can be used to supply blood to coronary artery CA downstreamof an occlusion, often using an end-to-side anastomosis coupling theharvested end of the IMA to an incision in the side of the occludedcoronary artery. To provide appropriate images to Operator O at mastercontrol station 200, the operator may sequentially select imagesprovided by either a first scope 306 a or a second scope 306 b forshowing on display 800 of the workstation. The camera switch procedurecan be understood through a description of an exemplary CABG procedurein which different camera views may be used. Two scopes are shown inFIG. 25A for illustrative purposes only. If only one image is desired,however, the procedure need not employ two endoscopes but instead needonly use one together with various instruments for actually performingthe procedure.

As seen in FIGS. 1 and 25A, it may generally be beneficial to accessheart H primarily through a pattern of apertures 930 disposed along aright side of patient P. Although the heart is primarily disposed in theleft side of the chest cavity, approaching the heart from the left sideof the chest as is typically done for MIS heart surgery may limit theamount of working volume available adjacent the target coronary tissues.This lack of working volume can complicate thoracoscopic roboticprocedures, as the lack of space can make it difficult to obtain apanoramic view of the heart surrounding the tissues targeted fortreatment, to quickly insert and remove tools, and to retract the heartappropriately for multi-vessel cases.

By inserting the elongate shafts of instruments 100 through the rightside of the patient, the apertures will be further away from the targetanatomy, including the left internal mammary artery (LIMA) and heart H.This approach can allow the camera to be separated from the targettissues by a greater distance, such as when a panorama or “big-picture”view is desired, while the resolution of robotic movement maintains thesurgeon's dexterity when the scope and tools extend across the chest tothe heart tissues for close-up views and work. The right-side approachmay also increase the speed with which tools can be changed, as theadditional separation between the aperture and the heart helps to ensurethat the heart is not in the way when delivering tools to harvest theIMA. When performing multi-vessel cases with the right-side approach,the heart can also be repeatedly retracted and repositioned so as tosequentially expose target regions of the heart to the significantworking volume available. Hence, different coronary vessels mayselectively be present to operator O for bypassing.

Advantageously, the right-side approach also facilitates dissection ofthe left internal mammary artery (LIMA) using a medial to lateralapproach. This dissection approach can provide a well-defined dissectionplane, can increase the ease with which branches can be seen, and mayprovide a view that is more familiar to surgeons accustomed totraditional CABG performed via a median sternotomy.

As can be seen most clearly in FIGS. 1 and 25B, cart 300 supports firstand second tools 100 a, 100 b for manipulating tissues (more may be usedbut are not shown) and first scope 306 a, while auxiliary cart 300Asupports second scope 306 b (and/or other manipulator tools, not shown).The arms of cart 300 preferably extend over the patient from thepatient's left side, and the instruments extend through aperture pattern930. The instrument shafts are generally angled to extend radiallyoutwardly from aperture pattern 930 in a “spoked wheel” arrangement tominimize interference between the manipulators. The exemplaryarrangement has scopes 306 a, 306 b extending through apertures definingthe top and bottom (anterior and posterior relative to the patient)positions of aperture pattern 930, while the manipulation tool shaftsdefine left and right (inferior and superior relative to the patient)positions. Second scope 306 b may be positioned through a lower, moredorsal aperture than shown, with the patient optionally being supportedon a table having an edge RE which is recessed adjacent aperture pattern930 to avoid interference between the auxiliary cart manipulator and thetable.

A robotic right-side approach CABG procedure might be outlined asfollows:

1. General anesthesia is initiated.

2. Patient is prepped in a basic supine position with a small roll underthe patient's scapula and back.

3. Patient is draped so that drapes start at about the posterioraxillary line.

LIMA Dissection:

4. Camera aperture for second scope 306 b is cut in an appropriateinnerspace (usually the 5^(th) intercostal space) on approximately theanterior axillary line. The first camera port may be positioned moremedially for directing anastomosis, and the like. If provided ordesired, the camera aperture for the first scope 306 a is cut in anappropriate innerspace (usually the 4^(th) or 5^(th) intercostal space)slightly posterior to the midclavicular line. Obviously, port placementfor the endoscopic tools as well as other portions of this procedure mayvary depending upon the anatomy of the particular patient in question.

5. Initiate insufflation at approximately 10 mm Hg.

6. Manipulation tool apertures are cut as appropriate (usually in the3^(rd) and 6^(th) or 7^(th) intercostal spaces for manipulationinstruments 100 a, 100 b) a few centimeters medial to the anteriorauxiliary line. Additional tool ports are placed as desired.

7. Robotic instruments 100 a, 100 b, 306 a, 306 b introduced throughapertures and robotic telesurgical control system is initiated.

8. LIMA harvesting is initiated by locating midline to establish thebeginning of dissection. Harvesting may be viewed and directed usingsecond scope 306 b.

9. LIMA is located by moving laterally using blunt dissection andcautery as desired.

10. Left pleura need not be entered, although insufflation can help keepleft lung out of the way.

RIMA Dissection (if desired)

11. The right internal mammary artery (RIMA) may be harvested usingsteps similar to 1-10 above, optionally through apertures disposed alongthe left side of the patient's chest.

Pericardiotomy

12. Incision may be made at surgeon's discretion. Preferably, anyincision will be high up on the pericardium, so that an attachment ofthe mediastinum to the chest forms a tent to enhance exposure of theheart.

IMA Preparation

13. IMA(s) may be prepared per surgeon's preference, typically whilestill attached to the chest.

Aorta Exposure

14. Aorta is exposed by extending the pericardiotomy cephalad asdesired. The pulmonary artery and any other adhesions may be dissectedoff so that the aorta can be clamped for cardio-pulmonary bypass and/orproximal grafts. As can be understood with reference to the descriptionabove of FIG. 23A, cardioplegia may be avoided by using a manual orrobotic cardiac tissue stabilizer mounted to, for example, auxiliarycart 300A during the anastomosis.

Coronary Artery Exposure

15. Target coronary artery or arteries can be exposed to working volumeby retracting and repositioning heart as desired, and standardtechniques may be used to expose and incise the exposed coronaryarteries.

Anastomosis

16. Anastomosis may be performed using needle-grasping tools 100 a, 100b while viewing display 800, as illustrated in FIG. 23A.

Suturing and exposure of the aorta and coronary artery or arteries mayat least in part be performed while viewing the moreanterior-to-posterior field of view provided from first scope 306 a, asmay portions of all other steps throughout the CABG procedure. When thesurgeon desires to change views between first and second image capturedevices, the surgeon may initiate the view change procedure byactivating a view change input device, possibly in the form of yetanother foot switch. The tissue manipulation tools will be briefly fixedin position, and the display will shift between the image capturedevices—for example, from the image provided from first scope 306 a, tothe image provided from second scope 306 b.

Optionally, the processor can reconfigure the coordinate transformationsbetween the masters and the end effectors when changing between twodifferent image capture devices to re-establish an at leastsubstantially connected relationship. This transformation modificationis similar to the process described above for a change in scopeposition, but will generally also accommodate the differences in supportstructure of the image capture devices. In other words, for example, themaster and/or slave kinematics 408, 412 (see FIG. 11) may be redefinedto maintain a correlation between a direction of movement of the inputdevice 210 and a direction of movement of an image of the end effector102 as shown in display 202 when viewing the end effector from adifferent scope. Similarly, when moving second scope 306 b (supported byauxiliary cart 300A) as a slave after a scope change from scope 306 a(which is supported by cart 300), the slave kinematics 412, slaveinput/output 414, and slave manipulator geometry 416 may all bedifferent, so that the control logic between the master and slave may berevised as appropriate.

More easily implemented approaches might allow the operator O to switchviews between scopes 306 a and 306 b without major software revisions.Using software developed to perform telesurgery with a single mastercontrol station 200 coupled to a single three arm cart 300 (see FIG. 1),switching the view to scope 306 b from scope 306 a might be accomplishedwhile maintaining the substantially connected relationship by “fooling”the processor of the master control station into believing that it isstill viewing the surgery through scope 306 a. More accurately, theprocessor may be fed signals which indicate that the middle set-up joint395 and/or manipulator arm 302 of cart 300 are supporting scope 306 a atthe actual orientation of scope 306 b. This may be accomplished bydecoupling the position sensing circuitry of the middle set-up jointand/or manipulator of cart 300 from the processor, and instead couplingan alternative circuit that transmits the desired signals. Thealternative “fooling” circuit may optionally be in the form of a sensorsystem of an alternative set-up joint and/or manipulator 302, whichmight be manually configured to hold a scope at the orientation of scope306 b relative to cart 300, but which need not actually supportanything. The image may then be taken from scope 306 b supported byauxiliary cart 300A, while the slave position signals x_(s) (See FIG.11) are taken from the alternative set-up joint. As described above, solong as the orientation of the end effectors relative to the scope areaccurately known, the system can easily accommodate positionalcorrections (such as by the translational clutching procedure describedabove).

Alternative telesurgical networks are schematically illustrated in FIGS.26 and 27. As mentioned above, an operator O and an Assistant A3 maycooperate to perform an operation by passing control of instrumentsbetween input devices, and/or by each manipulating their own instrumentor instruments during at least a portion of the surgical procedure.Referring, now to FIG. 26, during at least a portion of a surgicalprocedure, for example, cart 305 is controlled by Operator O andsupports an endoscope and two surgical instruments. Simultaneously, forexample, cart 308 might have a stabilizer and two other surgicalinstruments, or an instrument and another endoscope A3. The surgeon oroperator O and assistant A3 cooperate to perform a stabilized beatingheart CABG procedure by, for example, passing a needle or other objectback and forth between the surgical instruments of carts 305, 308 duringsuturing, or by having the instruments of cart 308 holding the tissue ofthe two vessels being anastomosed while the two instruments of cart 305are used to perform the actual suturing. Such cooperation heretofore hasbeen difficult because of the volumetric space required for human handsto operate. Since robotic surgical end effectors require much less spacein which to operate, such intimate cooperation during a delicatesurgical procedure in a confined surgical space is now possible.Optionally, control of the tools may be transferred or shared during analternative portion of the procedure.

Referring now to both FIGS. 26 and 27, cooperation between multiplesystems is also possible. The choice of how many masters and how manycorresponding slaves to enable on a cooperating surgical system issomewhat arbitrary. Within the scope of the present invention, one mayconstruct a single telesurgical system's architecture to handle five orsix manipulators (e.g., two masters and three or four slaves) or ten ortwelve manipulators (e.g., four masters and six or eight manipulators),although any number is possible. For a system having multiple mastercontrols, the system may be arranged so that two operators can operatethe same surgical system at the same time by controlling different slavemanipulators and swapping manipulators as previously described.

Alternatively, it may be desirable to have a somewhat modulartelesurgical system that is capable both of conducting one particularsurgical operation with only one operator and, for example, five or sixmanipulators, and which is also capable of coupling to another modularsystem having five or six manipulators to perform a second surgicalprocedure in cooperation with a second operator driving the secondsystem. For such modular systems, five or six manipulator arms arepreferably supported by the architecture, although any number may beincorporated into each system. One advantage of the modular system overa single, larger system is that when decoupled, the modular systems maybe used for two separate simultaneous operations at two differentlocations, such as in adjacent operating rooms, whereas such might bequite difficult with a single complex telesurgical system.

As can be understood with reference to FIG. 26, a simple manner ofhaving two surgical systems, each having an operator, to cooperateduring a surgical procedure is to have a single image capture device,such as an endoscope, produce the image for both operators. The imagecan be shared with both displays by using a simple image splitter. Ifimmersive display is desired, the two systems might additionally share acommon point of reference, such as the distal tip of the endoscope, fromwhich to calculate all positional movements of the slave manipulators,all as previously described in U.S. Appl. No. 60/128,160. With theexception of the imaging system, each control station might beindependent of the other, and might be operatively coupled independentlyto its associated tissue manipulation tools. Under such a simplecooperative arrangement, no swapping of slave manipulators from onesystem to another would be provided, and each operator would havecontrol over only the particular slave manipulators attached directly tohis system. However, the two operators would be able to pass certainobjects back and forth between manipulators, such as a needle during ananastomosis procedure. Such cooperation may increase the speed of suchprocedures once the operators establish a rhythm of cooperation. Such anarrangement scenario may, for example, be used to conduct a typical CABGprocedure, such that one operator would control the endoscope and twotissue manipulators, and the other operator would control two or threemanipulators to aid in harvesting the IMA and suturing the arterialblood source to the blocked artery downstream of the particular blockedartery in question. Another example where this might be useful would beduring beating heart surgery, such that the second operator couldcontrol a stabilizer tool in addition to two other manipulators andcould control the stabilizer while the first operator performed ananastomosis.

One complication of simple cooperative arrangements is that if the firstoperator desired to move the image capture device, the movement mightalter the image of the surgical field sufficiently that the secondoperator would no longer be able to view his slave manipulators. Thus,some cooperation between the operators, such as audible communications,might be employed before such a maneuver.

A slightly more complicated arrangement of surgical manipulators on twosystems within the scope of the present invention, occurs when operatorsare provided with the ability to “swap” control of manipulator arms. Forexample, the first operator is able to procure control over amanipulator arm that is directly connected to the second operator'ssystem. Such an arrangement is depicted in FIG. 26.

With the ability to operatively hook multiple telesurgical systemstogether, an arrangement akin to a surgical production line can beenvisioned. For example, a preferred embodiment of the present inventionis shown in FIG. 27. Therein, a single master surgeon O occupies acentral master control operating room. Satellite operating rooms (ORs)952, 954 and 956 are each operatively connected to the central masterconsole via switching assembly 958, which is selectively controlled byOperator O. While operating on a first patient P1 in OR 956, thepatients in ORs 954 and 952 are being prepared by assistants A2 and A3,respectively. During the procedure on patient P1, patient P3 becomesfully prepared for surgery, and A3 begins the surgery on the mastercontrol console dedicated to OR 952 by controlling manipulator assembly964. After concluding the operation in OR 956, Operator O checks with A3by inquiring over an audio communications network between the ORswhether A3 requires assistance. OR 950 might additionally have a bank ofvideo monitors showing the level of activity in each of the Ors, therebypermitting the master surgeon to determine when it would be best tobegin to participate in the various ongoing surgeries, or to handcontrol off to others to continue or complete some of the surgeries.

Returning to the example, if A3 requests assistance, O selects OR 952via switching assembly 958, selects a cooperative surgery set-up on anOR-dedicated switching assembly 960, and begins to control manipulatorassembly 962. After completion of the most difficult part of the surgeryin OR 952, O switches over to OR 954, where patient P2 is now ready forsurgery.

The preceding description is a mere example of the possibilities offeredby the cooperative coupling of masters and slaves and varioustelesurgical systems and networks. Other arrangements will be apparentto one of skill in the art reading this disclosure. For example,multiple master control rooms can be imagined in which several mastersurgeons pass various patients back and forth depending on theparticular part of a procedure being performed. The advantages ofperforming surgery in this manner are myriad. For example, the mastersurgeon O does not have to scrub in and out of every procedure. Further,the master surgeon may become extremely specialized in performing partof a surgical procedure, e.g., harvesting an IMA, by performing justthat part of a procedure over and over on many more patients than heotherwise would be able to treat. Thus, particular surgical procedureshaving distinct portions might be performed much more quickly by havingmultiple surgeons, with each surgeon each performing one part of theprocedure and then moving onto another procedure, without scrubbingbetween procedures. Moreover, if one or more patients (for whateverreason) would benefit by having a surgeon actually be present, analternative surgeon (different from the master surgeon) may be on callto one or more operating rooms, ready to jump in and address thepatient's needs in person, while the master surgeon moves on treatanother patient. Due to increased specialization, further advances inthe quality of medical care may be achieved.

In addition to enabling cooperative surgery between two or moresurgeons, operatively hooking two or more operator control stationstogether in a telesurgical networking system also may be useful forsurgical training. A first useful feature for training students orsurgeons how to perform surgical procedures would take advantage of a“playback” system for the student to learn from a previous operation.For example, while performing a surgical procedure of interest, asurgeon would record all of the video information and all of the dataconcerning manipulation of the master controls on a tangible machinereadable media. Appropriate recording media are known in the art, andinclude videocassette or Digital Video Disk (DVD) for the video imagesand/or control data, and Compact Disk (CD), e.g., for the servo datarepresenting the various movements of the master controls.

If two separate media are used to record the images and the servo data,then some method of synchronizing the two would be desirable duringfeedback, to ensure that the master control movements substantiallymirror the movements of the slave manipulators in the video image. Acrude but workable method of synchronization might include a simple timestamp and a watch. Preferably, both video images and servo data would berecorded simultaneously on the same recording medium, so that playbackwould be automatically synchronized.

During playback of the operation, a student could place his hands on themaster controls and “experience” the surgery, without actuallyperforming any surgical manipulations, by having his hands guided by themaster controls through the motions of the slave manipulators shown onthe video display. Such playback might be useful, for example, inteaching a student repetitive motions, such as during suturing. In sucha situation, the student would experience over and over how the mastersmight be moved to move the slaves in such a way as to tie sutures, andthus hopefully would learn how better to drive the telesurgical systembefore having to perform an operation.

The principles behind this playback feature can be built upon by using alive hand of a second operator instead of simple data playback. Forexample, two master control consoles may be connected together in such away that both masters are assigned to a single set of surgicalinstruments. The master controls at the subordinate console would followor map the movements of the masters at the primary console, but wouldpreferably have no ability to control any of the instruments or toinfluence the masters at the primary console. Thus, the student seatedat the subordinate console again could “experience” a live surgery byviewing the same image as the surgeon and experiencing how the mastercontrols are moved to achieve desired manipulation of the slaves.

An advanced version of this training configuration includes operativelycoupling two master consoles into the same set of surgical instruments.Whereas in the simpler version, one console was subordinate to the otherat all times, this advanced version permits both master controls tocontrol motion of the manipulators, although only one could controlmovement at any one time. For example, if the student were learning todrive the system during a real surgical procedure, the instructor at thesecond console could view the surgery and follow the master movements ina subordinate role. However, if the instructor desired to wrest controlfrom the student, e.g., when the instructor detected that the studentwas about to make a mistake, the instructor would be able to overridethe student operator by taking control over the surgical manipulatorsbeing controlled by the student operator. The ability to so interactwould be useful for a surgeon supervising a student or second surgeonlearning a particular operation. Since the masters on the instructor'sconsole were following the surgery as if he were performing it, wrestingcontrol is a simple matter of clutching into the surgery and overridingthe control information from the student console. Once the instructorsurgeon had addressed the issue, either by showing the student how toperform a certain part of the surgical procedure or by performing ithimself, the instructor could clutch out of the operation and permit thestudent to continue.

An alternative to this “on-off” clutching—whereby the instructor surgeonis either subordinate to the student or in command—would be a variableclutch arrangement. For example, again the instructor is subordinate tothe student's performance of a procedure, and has his masters follow themovement of the student's master controls. When the instructor desiresto participate in the procedure, but does not desire to wrest allcontrol from the student, the instructor could begin to exert somecontrol over the procedure by partially clutching and guiding thestudent through a certain step. If the partial control was insufficientto achieve the instructor's desired result, the instructor could thencompletely clutch in and demonstrate the desired move, as above.Variable clutching could be achieved by adjusting an input device, suchas a dial or a foot pedal having a number of discrete settingscorresponding to the percentage of control desired by the instructor.When the instructor desires some control, he or she could operate theinput device to achieve a setting of, for example, 50 percent control,in order to begin to guide the student's movements. Software could beused to calculate the movements of the end effectors based on thedesired proportionate influence of the instructor's movements over thestudent's. In the case of 50% control, for example, the software wouldaverage the movements of the two sets of master controls and then movethe end effectors accordingly, producing resistance to the student'sdesired movement, thereby causing the student to realize his error. Asthe surgeon desires more control, he or she could ratchet the inputdevice to a higher percentage of control, finally taking completecontrol as desired.

Other examples of hooking multiple telesurgical control stationstogether for training purposes will be apparent to one of skill in theart upon reading this disclosure. Although these training scenarios aredescribed by referring to real surgery, either recorded or live, thesame scenarios could be performed in a virtual surgical environment, inwhich, instead of manipulating the tissue of a patient (human or animal)cadaver, or model, the slave manipulators could be immersed, in avirtual sense, in simulation software. The software would then create asimulated virtual surgical operation in which the instructor and/orstudent could practice without the need for a live patient or anexpensive model or cadaver.

While the present invention has been described in some detail, by way ofexample and for clarity of understanding, a variety of changes,adaptation, and modifications will be obvious to those of skill in theart. For example and without limiting effect, robotic systems havingmore than four manipulators and/or more that two scopes may be provided.The manipulator arms can all be mounted to a single support base, ormight be arranged with two arms on each of two separate support bases.Hence, the scope of the present invention is limited solely by theappended claims.

1-43. (canceled)
 44. A surgical training method comprising: receiving afirst command from a first master control operated by a mentee;receiving a second command from a second master control operated by amentor; and commanding an object to be moved according to a percentageof control between the first and second commands, wherein the percentageof control is selectable by the mentor using an input device.
 45. Thesurgical training method of claim 44, further comprising: applying aresistance against movement of the first master control such that theresistance is indicative of a difference between the first command andthe commanded movement of the object.
 46. The surgical training methodof claim 44, further comprising: receiving an indication of thepercentage of control from the input device.
 47. The surgical trainingmethod of claim 46, wherein the input device comprises a dial having anumber of discrete settings, and wherein the receiving the percentage ofcontrol from the input device comprises: receiving an indication of adial setting from the dial.
 48. The surgical training method of claim46, wherein the input device comprises a foot pedal having a number ofdiscrete settings, and wherein the receiving the percentage of controlfrom an input mechanism operated by the mentor comprises: receiving anindication of a foot pedal setting from the foot pedal.
 49. A surgicaltraining method comprising: displaying movement of an end effector of asurgical instrument on a display screen viewable by a mentee as thesurgical instrument is being used to perform a medical procedure; andcommanding movement of a first master control being held by the mentee,wherein movement of the end effector is not responsive to movement ofthe first master control, and wherein the commanded movement of thefirst master control is synchronized with the movement of the endeffector being displayed on the display screen.
 50. The surgicaltraining method of claim 49, wherein the displaying movement of an endeffector of a surgical instrument on a display screen viewable by amentee as the surgical instrument is being used to perform a medicalprocedure comprises: playing a recorded video recording of the medicalprocedure on the display screen.
 51. The surgical training method ofclaim 49, wherein the displaying movement of an end effector of asurgical instrument on a display screen viewable by a mentee as thesurgical instrument is being used to perform a medical procedurecomprises: capturing images of the end effector of the surgicalinstrument as a mentor is commanding movement of the end effector byusing a second master control; and displaying the captured images on thedisplay screen.
 52. The surgical training method of claim 51, whereinthe commanding movement of a first master control being held by thementee comprises: commanding movement of the first master control tofollow movement of the second master control.